gistfile1.txt

❯ export CLUSTER_TYPE='pre-allocated'
❯ export REPLICAS=10
❯ export TEST_TYPE=expand-and-modify
❯ ./scale-test create; ./scale-test run; ./scale-test clean
Deploying EKS cluster. See configuration in /tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T19:46UTC/cluster-config.yaml
2025-02-06 14:46:26 [ℹ]  eksctl version 0.202.0-dev+5c8b046f6.2025-01-23T05:03:14Z
2025-02-06 14:46:26 [ℹ]  using region us-west-2
2025-02-06 14:46:27 [ℹ]  setting availability zones to [us-west-2d us-west-2b us-west-2c]
2025-02-06 14:46:27 [ℹ]  subnets for us-west-2d - public:192.168.0.0/19 private:192.168.96.0/19
2025-02-06 14:46:27 [ℹ]  subnets for us-west-2b - public:192.168.32.0/19 private:192.168.128.0/19
2025-02-06 14:46:27 [ℹ]  subnets for us-west-2c - public:192.168.64.0/19 private:192.168.160.0/19
2025-02-06 14:46:27 [ℹ]  nodegroup "pre-allocated-ng" will use "" [AmazonLinux2/1.32]
2025-02-06 14:46:27 [ℹ]  using Kubernetes version 1.32
2025-02-06 14:46:27 [ℹ]  creating EKS cluster "ebs-scale-pre-allocated" in "us-west-2" region with managed nodes
2025-02-06 14:46:27 [ℹ]  1 nodegroup (pre-allocated-ng) was included (based on the include/exclude rules)
2025-02-06 14:46:27 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2025-02-06 14:46:27 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --cluster=ebs-scale-pre-allocated'
2025-02-06 14:46:27 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "ebs-scale-pre-allocated" in "us-west-2"
2025-02-06 14:46:27 [ℹ]  CloudWatch logging will not be enabled for cluster "ebs-scale-pre-allocated" in "us-west-2"
2025-02-06 14:46:27 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-west-2 --cluster=ebs-scale-pre-allocated'
2025-02-06 14:46:27 [ℹ]  default addons vpc-cni, kube-proxy, coredns, metrics-server were not specified, will install them as EKS addons
2025-02-06 14:46:27 [ℹ]
2 sequential tasks: { create cluster control plane "ebs-scale-pre-allocated",
    2 sequential sub-tasks: {
        5 sequential sub-tasks: {
            1 task: { create addons },
            wait for control plane to become ready,
            associate IAM OIDC provider,
            no tasks,
            update VPC CNI to use IRSA if required,
        },
        create managed nodegroup "pre-allocated-ng",
    }
}
2025-02-06 14:46:27 [ℹ]  building cluster stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:46:27 [ℹ]  deploying stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:46:57 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:47:28 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:48:28 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:49:29 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:50:29 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:51:30 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:52:30 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:53:30 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:54:31 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-cluster"
2025-02-06 14:54:33 [ℹ]  creating addon
2025-02-06 14:54:34 [ℹ]  successfully created addon
2025-02-06 14:54:34 [!]  recommended policies were found for "vpc-cni" addon, but since OIDC is disabled on the cluster, eksctl cannot configure the requested permissions; the recommended way to provide IAM permissions for "vpc-cni" addon is via pod identity associations; after addon creation is completed, add all recommended policies to the config file, under `addon.PodIdentityAssociations`, and run `eksctl update addon`
2025-02-06 14:54:34 [ℹ]  creating addon
2025-02-06 14:54:35 [ℹ]  successfully created addon
2025-02-06 14:54:35 [ℹ]  creating addon
2025-02-06 14:54:35 [ℹ]  successfully created addon
2025-02-06 14:54:36 [ℹ]  creating addon
2025-02-06 14:54:36 [ℹ]  successfully created addon
2025-02-06 14:54:37 [ℹ]  creating addon
2025-02-06 14:54:37 [ℹ]  successfully created addon
2025-02-06 14:56:40 [ℹ]  addon "vpc-cni" active
2025-02-06 14:56:41 [ℹ]  deploying stack "eksctl-ebs-scale-pre-allocated-addon-vpc-cni"
2025-02-06 14:56:41 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-addon-vpc-cni"
2025-02-06 14:57:11 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-addon-vpc-cni"
2025-02-06 14:57:12 [ℹ]  updating addon
2025-02-06 14:57:22 [ℹ]  addon "vpc-cni" active
2025-02-06 14:57:23 [ℹ]  building managed nodegroup stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 14:57:24 [ℹ]  deploying stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 14:57:24 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 14:57:54 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 14:58:50 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 14:59:29 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 15:00:24 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 15:00:24 [ℹ]  waiting for the control plane to become ready
2025-02-06 15:00:25 [✔]  saved kubeconfig as "/Users/andsirey/.kube/config"
2025-02-06 15:00:25 [ℹ]  no tasks
2025-02-06 15:00:25 [✔]  all EKS cluster resources for "ebs-scale-pre-allocated" have been created
2025-02-06 15:00:26 [ℹ]  nodegroup "pre-allocated-ng" has 1 node(s)
2025-02-06 15:00:26 [ℹ]  node "ip-192-168-88-135.us-west-2.compute.internal" is ready
2025-02-06 15:00:26 [ℹ]  waiting for at least 1 node(s) to become ready in "pre-allocated-ng"
2025-02-06 15:00:26 [ℹ]  nodegroup "pre-allocated-ng" has 1 node(s)
2025-02-06 15:00:26 [ℹ]  node "ip-192-168-88-135.us-west-2.compute.internal" is ready
2025-02-06 15:00:26 [✔]  created 1 managed nodegroup(s) in cluster "ebs-scale-pre-allocated"
2025-02-06 15:00:26 [ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for pod identity association for service account "kube-system/ebs-csi-controller-sa",
        create pod identity association for service account "kube-system/ebs-csi-controller-sa",
    } }2025-02-06 15:00:26 [ℹ]  deploying stack "eksctl-ebs-scale-pre-allocated-podidentityrole-kube-system-ebs-csi-controller-sa"
2025-02-06 15:00:26 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-podidentityrole-kube-system-ebs-csi-controller-sa"
2025-02-06 15:00:57 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-podidentityrole-kube-system-ebs-csi-controller-sa"
2025-02-06 15:01:40 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-podidentityrole-kube-system-ebs-csi-controller-sa"
2025-02-06 15:01:41 [ℹ]  created pod identity association for service account "ebs-csi-controller-sa" in namespace "kube-system"
2025-02-06 15:01:41 [ℹ]  all tasks were completed successfully
2025-02-06 15:01:42 [ℹ]  kubectl command should work with "/Users/andsirey/.kube/config", try 'kubectl get nodes'
2025-02-06 15:01:42 [✔]  EKS cluster "ebs-scale-pre-allocated" in "us-west-2" region is ready
Deploying EBS CSI driver from chart /Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/../../charts/aws-ebs-csi-driver
NAME: aws-ebs-csi-driver
LAST DEPLOYED: Thu Feb  6 15:01:44 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
NOTES:
To verify that aws-ebs-csi-driver has started, run:

    kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"

[ACTION REQUIRED] Update to the EBS CSI Driver IAM Policy

Due to an upcoming change in handling of IAM polices for the CreateVolume API when creating a volume from an EBS snapshot, a change to your EBS CSI Driver policy may be needed. For more information and remediation steps, see GitHub issue #2190 (https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2190). This change affects all versions of the EBS CSI Driver and action may be required even on clusters where the driver is not upgraded.
                           PRE ebs-scale-karpenter-expand-and-modify-10-2025-02-05T20:10UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-10-2025-02-05T20:27UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-10-2025-02-06T19:09UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-100-2025-02-06T18:16UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-100-2025-02-06T18:28UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-100-2025-02-06T18:32UTC/
                           PRE ebs-scale-karpenter-expand-and-modify-1000-2025-02-06T17:59UTC/
                           PRE ebs-scale-karpenter-resize-and-modify-10-2025-02-04T23:10UTC/
                           PRE ebs-scale-karpenter-scale-sts-100-2025-01-30T21:51UTC/
                           PRE ebs-scale-karpenter-scale-sts-100-2025-02-04T16:37UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-100-2025-01-28T20:20UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-100-2025-02-04T14:45UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-100-2025-02-04T21:18UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-1000-2025-01-23T20:30UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-1000-2025-02-04T19:09UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-1000-2025-02-04T19:55UTC/
                           PRE ebs-scale-pre-allocated-scale-sts-1000-2025-02-04T21:05UTC/
Updating kubeconfig and restarting ebs-csi-controller Deployment
Updated context arn:aws:eks:us-west-2:208536326202:cluster/ebs-scale-pre-allocated in /Users/andsirey/.kube/config
deployment.apps/ebs-csi-controller restarted
Waiting for deployment "ebs-csi-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "ebs-csi-controller" successfully rolled out
Applying /Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/expand-and-modify-test/expand-and-modify.yaml. Exported to /tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/expand-and-modify.yaml
storageclass.storage.k8s.io/ebs-scale-test-expand-and-modify created
volumeattributesclass.storage.k8s.io/ebs-scale-test-expand-and-modify created
persistentvolumeclaim/ebs-claim-0 created
persistentvolumeclaim/ebs-claim-1 created
persistentvolumeclaim/ebs-claim-2 created
persistentvolumeclaim/ebs-claim-3 created
persistentvolumeclaim/ebs-claim-4 created
persistentvolumeclaim/ebs-claim-5 created
persistentvolumeclaim/ebs-claim-6 created
persistentvolumeclaim/ebs-claim-7 created
persistentvolumeclaim/ebs-claim-8 created
persistentvolumeclaim/ebs-claim-9 created
Wait for all PVCs to be bound
Patching PVCs with [{"op": "replace", "path": "/spec/volumeAttributesClassName", "value": "ebs-scale-test-expand-and-modify"},{"op": "replace", "path": "/spec/resources/requests/storage", "value": "2Gi"}]
persistentvolumeclaim/ebs-claim-0 patched
persistentvolumeclaim/ebs-claim-1 patched
persistentvolumeclaim/ebs-claim-4 patched
persistentvolumeclaim/ebs-claim-2 patched
persistentvolumeclaim/ebs-claim-3 patched
persistentvolumeclaim/ebs-claim-9 patched
persistentvolumeclaim/ebs-claim-7 patched
persistentvolumeclaim/ebs-claim-6 patched
persistentvolumeclaim/ebs-claim-8 patched
persistentvolumeclaim/ebs-claim-5 patched
Waiting until volumes modified and/or expanded
0/10 volumes modified
10/10 volumes modified
All volumes modified
6/10 volumes expanded
10/10 volumes expanded
All volumes expanded
Deleting resources
storageclass.storage.k8s.io "ebs-scale-test-expand-and-modify" deleted
volumeattributesclass.storage.k8s.io "ebs-scale-test-expand-and-modify" deleted
persistentvolumeclaim "ebs-claim-0" deleted
persistentvolumeclaim "ebs-claim-1" deleted
persistentvolumeclaim "ebs-claim-2" deleted
persistentvolumeclaim "ebs-claim-3" deleted
persistentvolumeclaim "ebs-claim-4" deleted
persistentvolumeclaim "ebs-claim-5" deleted
persistentvolumeclaim "ebs-claim-6" deleted
persistentvolumeclaim "ebs-claim-7" deleted
persistentvolumeclaim "ebs-claim-8" deleted
persistentvolumeclaim "ebs-claim-9" deleted
Waiting for all PVs to be deleted
No resources found
No PVs exist in the cluster, proceeding...
Port-forwarding ebs-csi-controller containers
Collecting metrics
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 3301 after 0 ms: Couldn't connect to server
Failed to collect metrics from port 3301, retrying after 5s...
Forwarding from 127.0.0.1:8084 -> 8084
Forwarding from [::1]:8084 -> 8084
Forwarding from 127.0.0.1:8082 -> 8082
Forwarding from [::1]:8082 -> 8082
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Forwarding from 127.0.0.1:3301 -> 3301
Forwarding from [::1]:3301 -> 3301
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 3301
100 13244    0 13244    0     0  41507      0 --:--:-- --:--:-- --:--:-- 41517
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 8081
100 64195    0 64195    0     0   159k      0 --:--:-- --:--:-- --:--:--  158k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 8082
100 14008    0 14008    0     0  42437      0 --:--:-- --:--:-- --:--:-- 42448
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 8084
100  8397    0  8397    0     0  34371      0 --:--:-- --:--:-- --:--:-- 34413
Generating clean version of exported data at /tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/cleaned_data.txt
/Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/collect-and-export-metrics.sh: line 72:  3461 Terminated: 15          kubectl port-forward "$CONTROLLER_POD_NAME" 3301:3301 -n kube-system
/Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/collect-and-export-metrics.sh: line 72:  3462 Terminated: 15          kubectl port-forward "$CONTROLLER_POD_NAME" 8081:8081 -n kube-system
/Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/collect-and-export-metrics.sh: line 72:  3463 Terminated: 15          kubectl port-forward "$CONTROLLER_POD_NAME" 8082:8082 -n kube-system
/Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/collect-and-export-metrics.sh: line 72:  3464 Terminated: 15          kubectl port-forward "$CONTROLLER_POD_NAME" 8084:8084 -n kube-system
Collecting ebs-plugin logs
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-resizer, liveness-probe
Collecting ebs-csi-controller Deployment and ebs-csi-node Daemonset yaml
Exporting everything in /tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC to S3 bucket s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-csi-controller.yaml to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-csi-controller.yaml
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/cleaned_data.txt to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/cleaned_data.txt
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-csi-node.yaml to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-csi-node.yaml
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/expand-and-modify.yaml to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/expand-and-modify.yaml
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/metrics.txt to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/metrics.txt
upload: ../../../../../tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-plugin-logs.txt to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/ebs-plugin-logs.txt
Metrics exported to s3://ebs-scale-tests-208536326202/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/
2025-02-06 15:03:32 [ℹ]  deleting EKS cluster "ebs-scale-pre-allocated"
2025-02-06 15:03:33 [ℹ]  will drain 0 unmanaged nodegroup(s) in cluster "ebs-scale-pre-allocated"
2025-02-06 15:03:33 [ℹ]  starting parallel draining, max in-flight of 1
2025-02-06 15:03:34 [ℹ]  deleted 0 Fargate profile(s)
2025-02-06 15:03:35 [✔]  kubeconfig has been updated
2025-02-06 15:03:35 [ℹ]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2025-02-06 15:03:38 [ℹ]
5 sequential tasks: { delete nodegroup "pre-allocated-ng", delete IAM OIDC provider, delete addon IAM "eksctl-ebs-scale-pre-allocated-addon-vpc-cni", deleting IAM resources stack "eksctl-ebs-scale-pre-allocated-podidentityrole-kube-system-ebs-csi-controller-sa", delete cluster control plane "ebs-scale-pre-allocated" [async]
}
2025-02-06 15:03:38 [ℹ]  will delete stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 15:03:38 [ℹ]  waiting for stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng" to get deleted
2025-02-06 15:03:38 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 15:04:08 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"
2025-02-06 15:04:42 [ℹ]  waiting for CloudFormation stack "eksctl-ebs-scale-pre-allocated-nodegroup-pre-allocated-ng"