setup guide of amazon-eks-pod-identity-webhook

commands.sh

#!/bin/env bash

# Install k3s
curl -sfL https://get.k3s.io | sh -

# Install Cert Manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

# Install webhook
git clone https://github.com/aws/amazon-eks-pod-identity-webhook.git
cd amazon-eks-pod-identity-webhook/
yum install -y golang jq
make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest

# Create Identity Provider (s3)
PRIV_KEY="sa-signer.key"
PUB_KEY="sa-signer.key.pub"
PKCS_KEY="sa-signer-pkcs8.pub"
ssh-keygen -t rsa -b 2048 -f $PRIV_KEY -m pem -N ''
ssh-keygen -e -m PKCS8 -f $PUB_KEY > $PKCS_KEY

export S3_BUCKET=test2023a
export AWS_REGION=us-east-1

_bucket_name=$(aws s3api list-buckets  --query "Buckets[?Name=='$S3_BUCKET'].Name | [0]" --out text)
if [ $_bucket_name == "None" ]; then
    if [ "$AWS_REGION" == "us-east-1" ]; then
        aws s3api create-bucket --bucket $S3_BUCKET
    else
        aws s3api create-bucket --bucket $S3_BUCKET --create-bucket-configuration LocationConstraint=$AWS_REGION
    fi
fi

echo "export S3_BUCKET=$S3_BUCKET"
export HOSTNAME=s3.$AWS_REGION.amazonaws.com
export ISSUER_HOSTPATH=$HOSTNAME/$S3_BUCKET

go run ./hack/self-hosted/main.go -key $PKCS_KEY  | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > keys.json

cat <<EOF > discovery.json
{
    "issuer": "https://$ISSUER_HOSTPATH",
    "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json",
    "authorization_endpoint": "urn:kubernetes:programmatic_authorization",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "claims_supported": [
        "sub",
        "iss"
    ]
}
EOF

aws s3 cp --acl public-read ./keys.json s3://$S3_BUCKET/keys.json
aws s3 cp --acl public-read ./discovery.json s3://$S3_BUCKET/.well-known/openid-configuration

mkdir -p /etc/rancher/k3s/
cat <<EOF > /etc/rancher/k3s/config.yaml
kube-apiserver-arg:
- --service-account-key-file=$PWD/sa-signer-pkcs8.pub
- --service-account-signing-key-file=$PWD/sa-signer.key
- --service-account-issuer=https://$ISSUER_HOSTPATH
#- --api-audiences=k3s
EOF
systemctl restart k3s

kubectl apply -f test.yaml

READMD.md

This is a setup guide for amazon-eks-pod-identity-webhook on self-hosted k3s cluster

• https://github.com/aws/amazon-eks-pod-identity-webhook/tree/master
• https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md

The goal in this example is to config the pod to have full S3 access to the AWS account.

Setup

AWS

S3

1. Create an S3 bucket.
2. Navigate to "Permissions" > "Block public access" and disable public ACL block.
3. Enable ACLs by going to "Permissions" > "Object OwnershipInfo".

Identity Provider

1. Go to IAM > Identity providers > Add provider.
2. Select "OpenID Connect".
3. Input "Provider Name" as your S3 bucket URL, for example, s3.us-east-1.amazonaws.com/test2023a.
4. Set "Audiences" as k3s.

IAM Role (for the pod. this role will be referenced in the sa config)

1. Create a new IAM role.
2. Name it as myrole.
3. Set up trusted entities.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "<use the ARN of created identity provider>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity"
        }
    ]
}

4. Attach AmazonS3FullAccess policy to the role.

k3s

Details mentioned in command.sh. This script will

1. Install k3s
2. Install cert manager.
3. Install webhook.
4. Create an identity provider with S3.

Examples:

discovery.json

{
    "issuer": "https://s3.us-east-1.amazonaws.com/test2023a",
    "jwks_uri": "https://s3.us-east-1.amazonaws.com/test2023a/keys.json",
    "authorization_endpoint": "urn:kubernetes:programmatic_authorization",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "claims_supported": [
        "sub",
        "iss"
    ]
}

keys.json

{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "dChUxQw2A6HpqZamkglk2Iq_paXdv-CB6j7-9MzbiZE",
      "alg": "RS256",
      "n": "uqELZImmnbm-X4IoYOctoW5FbPWE0wBYtTeYxJrOEIkVIXS48Q5dL8Zoiv-5y87dnWVRm1upIVqVV9funsGDTeUfUPMc8LgAp3yp1SvcMttM3s32HMRotd6VhkpryRENQeCn7HxtbFCDOWiQJr0TVV1l1u0neyveyX_XWqMOdNH7GN9S_AHzyERCCpH9oykzgzIES87rnevnY8y243DhbZHFMRW3FG2STYyNQ06KX3hz7624DXpDxvXHtAd8iJeva_xA6oBO7ZpF7zzPvs3lfFie2leubK_bCAjlNXnaa6hLxSUVTclHoRgXx_B0bmqkqImUCQY-Yed1ij6l8XlMKQ",
      "e": "AQAB"
    },
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "",
      "alg": "RS256",
      "n": "uqELZImmnbm-X4IoYOctoW5FbPWE0wBYtTeYxJrOEIkVIXS48Q5dL8Zoiv-5y87dnWVRm1upIVqVV9funsGDTeUfUPMc8LgAp3yp1SvcMttM3s32HMRotd6VhkpryRENQeCn7HxtbFCDOWiQJr0TVV1l1u0neyveyX_XWqMOdNH7GN9S_AHzyERCCpH9oykzgzIES87rnevnY8y243DhbZHFMRW3FG2STYyNQ06KX3hz7624DXpDxvXHtAd8iJeva_xA6oBO7ZpF7zzPvs3lfFie2leubK_bCAjlNXnaa6hLxSUVTclHoRgXx_B0bmqkqImUCQY-Yed1ij6l8XlMKQ",
      "e": "AQAB"
    }
  ]
}

4. Configure apiserver to use the identity provider.

Testing

Details are in test.sh.

1. Apply test.yaml to create test pod and sa. Make sure the webhook is running. If it's not, re-install the cert manager and webhook by re-running the command.

[root@node2 ~]# kubectl get po,sa
NAME                                       READY   STATUS    RESTARTS   AGE
pod/pod-identity-webhook-c4fb89547-rxdcz   1/1     Running   0          13m
pod/my-pod                                 1/1     Running   0          4m54s

NAME                                  SECRETS   AGE
serviceaccount/default                0         14m
serviceaccount/pod-identity-webhook   0         13m
serviceaccount/mysa                   0         4m54s

2. Install AWS CLI.

[root@node2 ~]# kubectl exec my-pod -- apk add aws-cli
OK: 143 MiB in 42 packages

3. Test it by listing the contents of the S3 bucket and get caller-identity.

[root@node2 ~]# kubectl exec my-pod -- aws sts get-caller-identity

{
    "UserId": "AROATKPGOJR2CXJBT2DPN:botocore-session-1683672493",
    "Account": "<ACCOUNT_ID>",
    "Arn": "arn:aws:sts::<ACCOUNT_ID>:assumed-role/myrole/botocore-session-1683672493"
}

[root@node2 ~]# kubectl exec my-pod -- aws s3 ls
2023-05-09 21:58:22 test2023a

4. If the call to other AWS services, it should fail.

[root@node2 ~]# kubectl exec my-pod -- aws ec2 describe-instances --region us-east-1

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.
command

test.sh

#!/bin/env bash

kubectl apply -f test.yaml
kubectl exec my-pod -- apk add aws-cli
kubectl exec my-pod -- aws sts get-caller-identity
kubectl exec my-pod -- aws s3 ls
kubectl exec my-pod -- aws ec2 describe-instances --region us-east-1

test.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysa
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT>:role/myrole"
    eks.amazonaws.com/audience: "k3s"
    eks.amazonaws.com/sts-regional-endpoints: "true"
    eks.amazonaws.com/token-expiration: "86400"

---

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: mysa
  containers:
  - name: alpine
    image: alpine:latest
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo 'Hello, World!'; sleep 10; done"]