Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
kube2iam a second manual

kube2iam your security partner

Introduction

The open source project kube2iam is Amazon aws specific. It allows your pod to have access to an aws role, without other pods on the same node having the same privledges. For example if your pod needs to use the aws cli command to copy / sync files with an s3 bucket you can use kube2iam in your cluster so that ONLY your pod has those rights.

Amazon has stated at Kubecon 2017 that their EKS solution ( Amazon hosted Kubernetes cluster), that they intend to use kube2iam to handle roles and permissions for pods.

Setup kube2iam in your cluster

To have a seamless deployment of kube2iam in your cluster there are some steps to follow that aren't exactly in the readme files. The person responsible for deploying the kube2iam software should deploy it as a daemonset in the cluster so that every node has it. Also until the development team or ops team is comfortable with kube2iam you should probably deploy it with the --verbose and --debug flags.

To insure that all the requests for credentials get caught, I personally would recommend using the iptables=true option. Althought from personal experience I have seen where removing this option or setting it to false doesn't remove the iptable entry. You actually have to restart the node. Finally if you are using a network overlay with Kubernetes ( if not you should be ), you need to tell it what host interface to use. Personally we use Calico.

My deployment file for kube2iam looks like this:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube2iam
  labels:
    app: kube2iam
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: kube2iam
    spec:
      hostNetwork: true
      serviceAccount: kube2iam
      containers:
        - image: jtblin/kube2iam:latest
          name: kube2iam
          args:
            - "--auto-discover-base-arn"
            - "--iptables=true"
            - "--host-ip=$(HOST_IP)"
            - "--host-interface=cali+"
            - "--verbose"
            - "--debug"
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 8181
              hostPort: 8181
              name: http
          securityContext:
            privileged: true

One final note to mention. If you are using RBAC with your Kubernetes cluster you are going to need to setup some Kubernete Roles and Service Accounts for kube2iam. I didn't find a good reference online so I created the following:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube2iam
  namespace: kube-system
---
apiVersion: v1
items:
  - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: kube2iam
    rules:
      - apiGroups: [""]
        resources: ["namespaces","pods"]
        verbs: ["get","watch","list"]
  - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: kube2iam
    subjects:
    - kind: ServiceAccount
      name: kube2iam
      namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: kube2iam
      apiGroup: rbac.authorization.k8s.io
kind: List

Setting up the Role in AWS

The Role

When you setup a role you are saying "I give permission for my software to have access to do X on Resource Y". A role is made up of a number of Amazon policies. For example here is the only policy attached to the role 'backup-2-s3':

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::backup"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::backup/*"
            ]
        }
    ]

I'm giving the user of this role the right to ListBuckets and to do whatever they want inside the s3 bucket named "backup". Let's assume that you've done this and you've deployed kube2iam with the above role and you can't get access to the s3 bucket. Well that's because you actually don't have permission to use this new role. You might have seen an error in the kube2iam logs like this:

time="2018-01-17T20:12:01Z" level=info msg="GET /latest/meta-data/iam/security-credentials/ (200) took 9281 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/ req.remote=100.106.175.8 res.duration=9281 res.status=200
time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.fobar.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8

Why? keep reading...

Trust Relationship

This is the topic that confused me for a long time. It wasn't until i sat down and worked through some examples did I really understand what was happening with the trust relationship. The online AWS documentation describes configuring and using a role as:

When you run commands using the role profile, the AWS CLI uses the source profile's credentials to call AWS Security Token Service and assume the specified role. The source profile must have permission to call sts:assume-role against the role, and the role must have a trust relationship with the source profile to allow itself to be assumed.

What does this mean??? We have 2 roles:

  1. The first role Role 1 is the kubernetes worker node role. This is the role that every worker node has associated to it in your Kubernetes cluster. This role has numerous policies attached to it that give the role the right to DescribeInstances and get certain directories in certain s3 buckets that kops uses to setup your nodes.
  2. Role 2 is the role that does the special work that you want, in this case to have access to a specific s3 bucket. This is the role that will be assumed.

You must allow Role 2 to be assumed by Role 1. So two additional steps must be done here:

  1. You have to modify Role 2 so that it can be Assumed by Role 1. You do this by modifying the trust relationship. You trust Role 2 to give it the action to assume Role 1.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::221044447717:role/nodes.k8s.baz.foobar.io"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
  1. You now need to add the policy to allow your Kubernete worker nodes to assume roles that are not in their default role. In kops you can do this as an additional policy. Or you can just attach a new policy to your existing node role:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::221044447717:instance-profile/backup-2-s3"
            ]
        }
    ]
}

One thing that you could do in this additional role, since you don't want to have to add additional fields every time you add a resource you want this node to assume, you could change the arn:aws:iam::221044447717:instance-profile/backup-2-s3 to a *. That way you wouldn't have to edit it every time you want to add a new role for that node to assume.

How to define your deployments

Using kube2iam in your cluster is actually pretty easy. In your deployment file have an annotation in the metadata that specifies the name of the role that you want to use. For example to use my backup s3 role I would add the line iam.amazonaws.com/role: backup-2-s3 under annotations:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: remote
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: remote
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: remote
      annotations:
        iam.amazonaws.com/role: backup-2-s3
    spec:
      containers:
      - name: remote
        image: "iotapi322/worker:v4"
        resources:
          limits:
            cpu: "200m"
            memory: "256Mi"
          requests:
            cpu: "100m"
            memory: "128Mi"

Success!

To show if you truly hvae access to the s3 bucket you can do two things.

  1. Check if you get the correct role value back
  2. Try to use the resource

Well after the above steps, I verified that I was getting the correct role back:

root@remote-79cdc8dd46-mg2g5:/# curl 169.254.169.254/latest/meta-data/iam/security-credentials/
backup-2-S3

and I verfied that I could list the contents of my s3 bucket:

root@remote-79cdc8dd46-mg2g5:/# aws s3 ls s3://backup/EFSshares/
                           PRE api-adventurebot-01/
                           PRE api-adventurebot-02/
                           PRE hello-world-nginx/
                           PRE lockbot-db2/
                           PRE lockbot/
                           PRE test-permissions/
                           PRE test/

Troubleshooting

Troubleshooting permissions and Roles in AWS can be frustrating, however I've found that the logs from kube2iam to be really quite good. When in doubt find the node that your pod is running on. Next find the kube2iam pod that is also running on that node and start tailing the log.

Basic Debugging

You can curl inside your pod to the meta-data ip address to find out what your role is. If you have specified an annotation and you don't get that annotation your deployment is probably setup wrong.

Send your curl request set to: curl 169.254.169.254/latest/meta-data/iam/security-credentials/ (you have to have the trailing slash), you should get returned the name of the role that you placed in the annotation. If not something has gone awry.

I would suggest going into the kubernetes dashboard and finding which node your container is running on, then find the kube2iam pod that is running on that same node and looking at the kube2iam logs.

Error Msg's...

If you don't have the trust relationship in place you might see the following log lines from the kube2iam pod

time="2018-01-18T16:12:39Z" level=info msg="GET /latest/meta-data/iam/security-credentials (301) took 802140 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials req.remote=100.96.7.199 res.duration=802140 res.status=301
time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.tropo.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8

The highlights from the above error blog are:

i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3.

This is showing that the trust relationship has not been set on the worker node.

Another example

When you remote into your pod and run the aws cli command aws s3 ls s3://<s3-bucket>/ you will get back an error like this:

Unable to locate credentials. You can configure credentials by running "aws configure".

This is because you don't have a trust relationship setup between the kubernetes node and your role

Conclusion

I hope this helps clear up some of the confusion with Roles and Trust Relationships. kube2iam is a very powerful and necessary tool in your InfoSec toolbox.

More Resources:

@lambertpan

This comment has been minimized.

Copy link

lambertpan commented Jul 10, 2018

Hi,
I followed the steps you mentioned, but unable to get it working. I get the log max up-to;

level=debug msg="Pod OnUpdate" pod.iam.role="arn:aws:iam::........
time="2018-07-10T11:28:17Z" level=debug msg="Namespace OnUpdate" ns.name=kube-public
time="2018-07-10T11:28:17Z" level=debug msg="Namespace OnUpdate" ns.name=kube-system
time="2018-07-10T11:28:17Z" level=debug msg="Namespace OnUpdate" ns.name=default

But I don't get something like 'AccessDenied' - I am not sure where to look the issue, or is just an issue with kube2iam? . Any idea?

@jordan-huangwei

This comment has been minimized.

Copy link

jordan-huangwei commented Aug 8, 2018

Hi,
I meet the following problem:

lime="2018-08-08T08:20:54Z" level=info msg="GET /latest/meta-data/iam/security-credentials/ (200) took 37291 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/ req.remote=10.100.210.245 res.duration=37291 res.status=200
time="2018-08-08T08:20:54Z" level=error msg="Error assuming role InvalidClientTokenId: The security token included in the request is invalid\n\tstatus code: 403, request id: f81688f5-9ae3-11e8-8120-09b624201a66" ns.name=dev pod.iam.role="arn:aws-cn:iam::624768952373:role/k8s_access-s3-bucket-role" req.method=GET req.path=/latest/meta-data/iam/security-credentials/k8s_access-s3-bucket-role req.remote=10.100.210.245
time="2018-08-08T08:20:54Z" level=info msg="GET /latest/meta-data/iam/security-credentials/k8s_access-s3-bucket-role (500) took 254977679 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/k8s_access-s3-bucket-role

I don't know what cause this problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.