-
-
Save vfarcic/f0d35ef2260208b15ddd390007fdd552 to your computer and use it in GitHub Desktop.
| # Source: https://gist.github.com/f0d35ef2260208b15ddd390007fdd552 | |
| ###################################################################### | |
| # Production-Ready Kubernetes Clusters Using Crossplane Compositions # | |
| # https://youtu.be/uMC2QQfMctg # | |
| ###################################################################### | |
| # Referenced videos: | |
| # - Crossplane - GitOps-based Infrastructure as Code through Kubernetes API: https://youtu.be/n8KjVmuHm7A | |
| # - How To Shift Left Infrastructure Management Using Crossplane Composites: https://youtu.be/AtbS1u2j7po | |
| ################# | |
| # Setup Cluster # | |
| ################# | |
| # Watch https://youtu.be/BII6ZY2Rnlc if you are not familiar with GitHub CLI | |
| gh repo fork vfarcic/devops-toolkit-crossplane \ | |
| --clone | |
| cd devops-toolkit-crossplane | |
| # Using Rancher Desktop for the demo, but it can be any other Kubernetes cluster with Ingress | |
| # If not using Rancher Desktop, replace `127.0.0.1` with the base host accessible through NGINX Ingress | |
| export INGRESS_HOST=127.0.0.1 | |
| kubectl create namespace crossplane-system | |
| kubectl create namespace a-team | |
| ############# | |
| # Setup AWS # | |
| ############# | |
| # Replace `[...]` with your access key ID` | |
| export AWS_ACCESS_KEY_ID=[...] | |
| # Replace `[...]` with your secret access key | |
| export AWS_SECRET_ACCESS_KEY=[...] | |
| echo "[default] | |
| aws_access_key_id = $AWS_ACCESS_KEY_ID | |
| aws_secret_access_key = $AWS_SECRET_ACCESS_KEY | |
| " >aws-creds.conf | |
| kubectl --namespace crossplane-system \ | |
| create secret generic aws-creds \ | |
| --from-file creds=./aws-creds.conf | |
| #################### | |
| # Setup Crossplane # | |
| #################### | |
| helm repo add crossplane-stable \ | |
| https://charts.crossplane.io/stable | |
| helm repo update | |
| helm upgrade --install crossplane crossplane-stable/crossplane \ | |
| --namespace crossplane-system --create-namespace --wait | |
| kubectl apply \ | |
| --filename crossplane-config/provider-kubernetes.yaml | |
| kubectl apply \ | |
| --filename crossplane-config/provider-helm.yaml | |
| kubectl apply \ | |
| --filename crossplane-config/config-k8s.yaml | |
| kubectl get pkgrev | |
| # Wait until all packages are healthy | |
| kubectl apply \ | |
| --filename crossplane-config/provider-config-aws-official.yaml | |
| ############################# | |
| # Production-Ready Clusters # | |
| ############################# | |
| cat examples/k8s/aws-eks-official.yaml | |
| kubectl --namespace a-team apply \ | |
| --filename examples/k8s/aws-eks-official.yaml | |
| kubectl --namespace a-team get clusterclaims | |
| kubectl get managed,releases,objects | |
| cat packages/k8s/definition.yaml | |
| cat packages/k8s/eks-official.yaml | |
| kubectl get managed,releases,objects | |
| kubectl --namespace a-team \ | |
| get secret a-team-eks-cluster \ | |
| --output jsonpath="{.data.kubeconfig}" \ | |
| | base64 -d >kubeconfig.yaml | |
| cat kubeconfig.yaml | |
| export KUBECONFIG=$PWD/kubeconfig.yaml | |
| kubectl get namespaces | |
| cat examples/k8s/aws-eks.yaml | |
| ########### | |
| # Destroy # | |
| ########### | |
| unset KUBECONFIG | |
| kubectl --namespace a-team delete \ | |
| --filename examples/k8s/aws-eks-official.yaml | |
| kubectl get managed | |
| # Repeat the previous command until all the managed resources | |
| # are removed (ignore `object` and `release` resources`) | |
| # Destroy or reset the management cluster | |
| # Destroy the GitOps repo |
Did you wait a bit after executing kubectl apply --filename crossplane-config/provider-aws.yaml. The crossplane-config/provider-config-aws.yaml depends on crossplane-config/provider-aws.yaml being up-and-running.
If you did wait for a while, please execute kubectl get pkgrev and paste the output.
I did wait, and kubectl get pkgrev finds no resources.
devops-toolkit-crossplane]$ kubectl get pkgrev
No resources found
Also, just to make sure,
devops-toolkit-crossplane]$ kubectl apply --filename crossplane-config/provider-aws.yaml
provider.pkg.crossplane.io/crossplane-provider-aws unchanged
Thanks for replying. I'm on the crossplane slack if that's a better place to discuss.
Can you ping me on Slack (@vfarcic)? I found Jason but I'm not sure that's you.
Hi Victor!
Amazing effort and videos, really helpful for many of us, thank you!
I want to let you know that current master contains probably an issue or is it intentional? Let me explain...
After kubectl --namespace a-team delete --filename examples/k8s/aws-eks.yaml and after everything is properly removed from Crossplane and Amazon PoV, there is still existing created VPC. Removing it from AWS WebGUI doesn't make sense because k8s and desired state, removing from Crossplane seems tricky - I tried just to kubectl delete vpc.ec2.aws.crossplane.io/a-team-eks but it was re-created again. Ok, so I've tried to remove all 'spare' objects - in my case kubectl delete object.kubernetes.crossplane.io/a-team-eks-schemahero-ns object.kubernetes.crossplane.io/a-team-eks-aws-pc providerconfigusage.kubernetes.crossplane.io/8fc9a828-e5a0-4e0e-8d4a-fd5f48a7aaf1 vpc.ec2.aws.crossplane.io/a-team-eks providerconfigusage.aws.crossplane.io/c536c9e8-42f3-4abf-a143-2e9efe69447a - hm, call hangs but VPC still exists... nice, so I've deleted the team NS kubectl delete ns a-team without an issue however those 5 objects survived... so repeated previous deletion of 5 objects - even the call hangs again and (almost) all objects still exist, the VPC was deleted at least which was my intention (existing VPC is charged money). So now I don't have a-team NS however still have (from kubectl get crossplane):
NAME AGE CONFIG-NAME RESOURCE-KIND RESOURCE-NAME
providerconfigusage.kubernetes.crossplane.io/8fc9a828-e5a0-4e0e-8d4a-fd5f48a7aaf1 10h a-team-eks Object a-team-eks-aws-pc
providerconfigusage.kubernetes.crossplane.io/b9094b7f-bde1-4007-bbed-4e6ac8aa7e77 10h a-team-eks Object a-team-eks-schemahero-ns
NAME SYNCED READY AGE
object.kubernetes.crossplane.io/a-team-eks-aws-pc False False 14h
object.kubernetes.crossplane.io/a-team-eks-schemahero-ns False False 14h
The rest (not pasted above) are 'system' objects seem not related to the created ClusterClaim. I don't see that as a correct final state but can live with it anyway...
Any idea/advise?
Pavel
The problem is likely with the Load Balancer. When you install Ingress inside a cluster, it creates an external LB which, in turn, it tied to a VPC. As a result, the VPC cannot be deleted as long as that LB is there and, since it was not created by Crossplane, it is unaware of it. You'll experience the same behavior with other tools (e.g., Terraform).
For now, the workaround is to delete the LoadBalancer Service. Something like... kubectl --namespace ingress-nginx delete service a-team-eks-ingress-ingress-nginx-controller.
Hi Victor,
thank you for the hint with LBS, seems forking since then.
It seems there is an other issue (or quite recent change) regarding SecurityGroup definition here. In your case when you specify 'All protocols' and use fromPort and toPort, such an object has never be SYNC=true. The work-around for this might be remove fromPort,toPort and use just ipProtocol: "-1" and ipRanges eventualy. The same might be applied for ingress if necessary. By this way you are able to create a new SecurityGroup similarly like the default EKS SecGroup created by Amazon themself.
Not sure if it is intentional or a bug which should be reported - if an issue isn't reported in GitHub yet....
That's not intentional but, rather, an oversight on my part. Would you like to create a PR with the changes?
Sure, happy to contribute, you have helped me much through you excellent videos already! (PR)
Couple of notes:
- The same
SecurityGroupdefinition you are using in other demos/repositories as far as I can see - Don't forget to build configuration and then update reference here (and everywhere where you are referencing the same cfg.:-D)
- Not sure if the same work-around can be applied to
SecurityGroupRule(there you use similaregresstoo) - e.g.eks-official.yaml. I've never need to use Rule(s) yet, needs to be tested first
Anyway - this is some kind of orphan? I don't see it in a quite recent file eks-official.yaml but can be seen also in gke-official.yaml...
Thanks a ton for the PR.
- That's what happens when you make a mistake and then replicate that same mistake in other places without reevaluating the code. Unfortunately, that's an explanation, not an excuse (there is none). I don't have any :(
- Done :)
- I can fix and test that one the next time I'm working on that Composition.
this is there only because some of the uses of that one required the key in the secret to being different (to be value). I think that was added for SchemaHero and, in the meantime, I figured out how to use it without that key. It can be removed but, to do that, I need to find all the Gists that use it and modify them first. That's the type of issue I'm experiencing all the time. I create a lot of public Gists, demos, slides, books, etc. and I'm doing my best to keep them always in the working state. The downside is that any change that might not be backward compatible requires going back to all those Gists, demos, slides, etc.
Hi
I am getting below error even though my provider is showing in true and healthy state below is my output.I followed the same process to create a cluster.
kubectl get pkgrev
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
configurationrevision.pkg.crossplane.io/crossplane-k8s-61efd4c23f27 True 1 xpkg.upbound.io/devops-toolkit/dot-kubernetes:v0.9.23 Active 2 2 10m
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
providerrevision.pkg.crossplane.io/crossplane-provider-aws-a2e16ca2fc1a True 1 crossplane/provider-aws:v0.29.0 Active 14m
providerrevision.pkg.crossplane.io/crossplane-provider-helm-59fc94cd939e True 1 xpkg.upbound.io/crossplane-contrib/provider-helm:v0.13.0 Active 11m
providerrevision.pkg.crossplane.io/crossplane-provider-kubernetes-05560a2c7b16 True 1 xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.6.0 Active 10m
kubectl get managed,object,release
Warning: Please use v1beta1 version of SNS group.
NAME READY SYNCED ID VPC CIDR AGE
subnet.ec2.aws.crossplane.io/a-team-eks-1a False 10.0.0.0/24 2m40s
subnet.ec2.aws.crossplane.io/a-team-eks-1b False 10.0.1.0/24 2m40s
subnet.ec2.aws.crossplane.io/a-team-eks-1c False 10.0.2.0/24 2m40s
NAME READY SYNCED ID VPC AGE
internetgateway.ec2.aws.crossplane.io/a-team-eks False 2m40s
NAME READY SYNCED ID VPC AGE
routetable.ec2.aws.crossplane.io/a-team-eks False 2m40s
NAME READY SYNCED ID VPC AGE
securitygroup.ec2.aws.crossplane.io/a-team-eks False 2m40s
NAME READY SYNCED ID CIDR IPV6CIDR AGE
vpc.ec2.aws.crossplane.io/a-team-eks False False 10.0.0.0/16 2m40s
NAME READY SYNCED AGE
cluster.eks.aws.crossplane.io/a-team-eks False 2m40s
NAME READY SYNCED CLUSTER AGE
nodegroup.eks.aws.crossplane.io/a-team-eks False 2m40s
NAME CHART VERSION SYNCED READY STATE REVISION DESCRIPTION AGE
release.helm.crossplane.io/a-team-eks-crossplane crossplane 1.9.0 False 2m40s
release.helm.crossplane.io/a-team-eks-ingress ingress-nginx 4.2.0 False 2m40s
NAME READY SYNCED ROLENAME POLICYARN AGE
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-cni False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-controlplane False a-team-eks-controlplane arn:aws:iam::aws:policy/AmazonEKSClusterPolicy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-registry False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-service False a-team-eks-controlplane arn:aws:iam::aws:policy/AmazonEKSServicePolicy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-worker False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy 2m40s
NAME READY SYNCED AGE
role.iam.aws.crossplane.io/a-team-eks-controlplane False 2m40s
role.iam.aws.crossplane.io/a-team-eks-nodegroup False 2m40s
NAME KIND PROVIDERCONFIG SYNCED READY AGE
object.kubernetes.crossplane.io/a-team-eks-aws-p Provider a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-aws-pc ProviderConfig a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-app Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-monitoring Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-sql Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-helm-provider Provider a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-cc ControllerConfig a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-crb ClusterRoleBinding a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-sa ServiceAccount a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-kubernetes-provider Provider a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-ns-dev Namespace a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-ns-prod Namespace a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-schemahero-cr ClusterRole a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crb ClusterRoleBinding a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-db CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-migration CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-table CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-ns Namespace a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-secret Secret a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-service Service a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-sts StatefulSet a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-sql-provider Provider a-team-eks False 2m39s
Given that all the resources (except providers and configs) are False, I'm guessing that there's something wrong with the authentication to AWS. Can you describe one of the resources and post it here? Some of them are probably False because they depend on others that were not created so describing one of those that can be created independently of others. So, let's see what are the events of, let's say, the VPC. Something like kubectl describe vpc.ec2.aws.crossplane.io/a-team-eks should do.
create failed: failed to create the VPC resource: api error UnauthorizedOperation: You are not authorized to perform this operation getting this error even though I have assume the role which has access to create resources
I'm guessing that the creds you used starting from https://gist.github.com/vfarcic/f0d35ef2260208b15ddd390007fdd552#file-130-crossplane-k8s-helm-sh-L35 are invalid or do not have the permissions to create, in this case, the VPC. Can you confirm that the access key ID put into the env. var. AWS_ACCESS_KEY_ID and the secret access key stored in AWS_SECRET_ACCESS_KEY have the permissions and that the commands after that line did create the secret with those keys?
Yes my access_key and secret_key has the required permission to create or delete the resources. I am using cross-account in aws . Do i need to specify the specific role it requires to create resources
I'm not sure what the exact cause of the issue is, but I am fairly certain that those keys alone do not give Crossplane sufficient permissions to create AWS resources. Check the ProviderConfig spec (you can find it in https://marketplace.upbound.io) and see whether something is missing.
Hi @vfarcic
Any idea about how my kube config should look like for my k8s provider? Do you have a sample so that the k8s provider can access remote eks cluster?
crossplane-contrib/provider-kubernetes#121
@ashishvaishno You need to install Kubernetes provider and create a secret with KubeConfig pointing to the cluster you want to operate through Crossplane. With those two up and running, you need to create ProviderConfig that points to that secret. I don't have an example of that since I tend to create those clusters with Crossplane Compositions which, among other things, do all that for me. Here's one example of such a composition:
https://github.com/vfarcic/crossplane-kubernetes/blob/main/package/civo-port.yaml
In that example, I already have Kubernetes provider installed in the control plane cluster. From there on, https://github.com/vfarcic/crossplane-kubernetes/blob/main/package/civo-port.yaml#L80 is the part that configures the provider to use the secret previously generated by that same composition.
Please let me know if you have difficulties replicating that and I'll do my best to create a better example.
@vfarcic. Thanks for your instant reply 😄 In my case both the eks clusters were created without using crossplane. i was looking at your tutorial https://www.youtube.com/watch?v=uMC2QQfMctg&t=349s, where your kube config has a token for the user. As you also mentioned in the video that the token is short lived, how can I create a kubeconfig in such case if the remote cluster is not managed by crossplane?
@ashishvaishno If the cluster is not managed by Crossplane, you'd need to fetch the config without Crossplane (e.g., with aws CLI). From there on, you either give it to others or, if it's for Crossplane usage, create a secret with it. You might also be able to fetch the config with Observe Only Resources (https://github.com/crossplane/crossplane/blob/master/design/design-doc-observe-only-resources.md) but I'm not sure whether that would work (I haven't tried it with Kube config myself).
I'd probably start with OOR first and, if that does not work, fall back to aws.
the aws cli doesnt seems to be present in the kubernetes provider pod 😢
2023-05-31T11:13:06.648Z DEBUG provider-kubernetes Cannot connect to provider {"controller": "managed/object.kubernetes.crossplane.io", "request": "/xxxxxx-qbx82-wx525", "uid": "a26bad78-97ee-4af4-bc53-7c4cc33b965a", "version": "5789603", "external-name": "xxxxxx-qbx82-wx525", "error": "cannot create new Kubernetes client: cannot create Kubernetes client: Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins", "errorVerbose": "Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins\ncannot create Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/clients.NewKubeClient\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/clients/client.go:38\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:174\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594\ncannot create new Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:176\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
let me try the observe-only-resource 😄
@vfarcic I installed the latest version of crossplane, but it doesnt have the alpha feature flag --enable-alpha-management-policies 😕
k exec -it -n crossplane-system crossplane-5697555f88-qtl9h -c crossplane -- crossplane --version
v1.12.1
k exec -it -n crossplane-system crossplane-5697555f88-qtl9h -c crossplane -- crossplane core start --help
Usage: crossplane core start
Start Crossplane controllers.
Flags:
-h, --help Show context-sensitive help.
-d, --debug Print verbose logging statements.
-v, --version Print version and quit.
--profile=host:port Serve runtime profiling data via HTTP at /debug/pprof.
-n, --namespace="crossplane-system" Namespace used to unpack and run packages ($POD_NAMESPACE).
--service-account="crossplane" Name of the Crossplane Service Account ($POD_SERVICE_ACCOUNT).
-c, --cache-dir="/cache" Directory used for caching package images ($CACHE_DIR).
-l, --leader-election Use leader election for the controller manager ($LEADER_ELECTION).
-r, --registry="index.docker.io" Default registry used to fetch packages when not specified in tag ($REGISTRY).
--ca-bundle-path=STRING Additional CA bundle to use when fetching packages from registry ($CA_BUNDLE_PATH).
--webhook-tls-secret-name=STRING The name of the TLS Secret that will be used by the webhook servers of core Crossplane and providers ($WEBHOOK_TLS_SECRET_NAME).
--webhook-tls-cert-dir=STRING The directory of TLS certificate that will be used by the webhook server of core Crossplane. There should be tls.crt and tls.key files ($WEBHOOK_TLS_CERT_DIR).
--user-agent="crossplane/v1.12.1" The User-Agent header that will be set on all package requests ($USER_AGENT).
-s, --sync-interval=1h How often all resources will be double-checked for drift from the desired state.
--poll-interval=1m How often individual resources will be checked for drift from the desired state.
--max-reconcile-rate=10 The global maximum rate per second at which resources may checked for drift from the desired state.
--esstls-secret-name=STRING The name of the TLS Secret that will be used by Crossplane and providers as clients of External Secret Store plugins ($ESS_TLS_SECRET_NAME).
--esstls-certs-dir=STRING The path of the folder which will store TLS certificates to be used by Crossplane and providers for communicating with External Secret Store plugins ($ESS_TLS_CERTS_DIR).
Alpha Features:
--enable-environment-configs Enable support for EnvironmentConfigs.
--enable-external-secret-stores Enable support for External Secret Stores.
--enable-composition-functions Enable support for Composition Functions.
--enable-composition-webhook-schema-validation Enable support for Composition validation using schemas.
k describe po -n crossplane-system crossplane-5697555f88-qtl9h | grep -i image
Image: crossplane/crossplane:v1.12.1
@ashishvaishno I probably misunderstood your scenario so let me say what I understood and you correct me.
You have Kubernetes clusters created somehow (NOT with Crossplane) and now you'd like to use Crossplane to manage resources in those clusters. To do that, you need to configure the Crossplane Kubernetes provider with authentication to all those clusters and for that, you need Kube config stored as a secret inside the cluster where Crossplane is running (if those were created with Crossplane you would already have those secrets). So, you need to fetch Kube configs whichever way you would normally fetch them and create (yourself) Kubernetes secrets with those configs. From there on, Crossplane will know how to interact with those clusters and do whatever you'd want it to do.
Now, if I'm right in those assumptions (and I probably am not), all you have to do is 1) fetch Kube configs from your laptop or any other machine that has aws CLI, 2) create secrets with those configs, and 3) create Kubernetes provider configs that point to those secrets. Now, bear in mind that you should not store manifests of those secrets in Git. That would be unsafe. Either you create secrets and delete the manifests from your laptop (or wherever you created them from) or you use something like External Secrets Operator (ESO) or Sealed Secrets.
@vfarcic Below is the rough interpretation of my architecture
- I have provisioned a new eks cluster(with terraform) and installed crossplane on it. I have also installed the k8s provider there.
- I already had an old eks cluster(also provisioned with terraform) with different application works loads (cross-account-eks-cluster)
- For my POC, I would like to use crossplane to provision new resources on the cross-account-eks-cluster
- My config looks like this
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxx (ca of cross-account-eks-cluster)
server: xxxxxx (url from cross-account-eks-cluster)
name: cross-account-eks-cluster
contexts:
- context:
cluster: cross-account-eks-cluster
user: cross-account-eks-cluster
name: cross-account-eks-cluster
current-context: cross-account-eks-cluster
kind: Config
preferences: {}
users:
- name: cross-account-eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- xxxxx
- eks
- get-token
- --cluster-name
- cross-account-eks-cluster
- I pass this as a base 64 encoded secret to my kubernetes ProviderConfig, (I am not going to store it in git 😉)
---
apiVersion: v1
kind: Secret
metadata:
name: cluster-config
namespace: crossplane-system
data:
kubeconfig-dev: xxxxxxx
---
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: kubernetes-provider-config-dev
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: cluster-config
key: kubeconfig-dev
- Now while trying to create a namespace on the cross-account-eks-cluster I get the below error from the k8s-provider-pod as aws-cli is not installed in it 😢
2023-05-31T11:13:06.648Z DEBUG provider-kubernetes Cannot connect to provider {"controller": "managed/object.kubernetes.crossplane.io", "request": "/xxxxxx-qbx82-wx525", "uid": "a26bad78-97ee-4af4-bc53-7c4cc33b965a", "version": "5789603", "external-name": "xxxxxx-qbx82-wx525", "error": "cannot create new Kubernetes client: cannot create Kubernetes client: Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins", "errorVerbose": "Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins\ncannot create Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/clients.NewKubeClient\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/clients/client.go:38\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:174\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594\ncannot create new Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:176\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v0.19.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
- I also tried the observe only feature, but I dont find that feature flag while installing crossplane https://gist.github.com/vfarcic/f0d35ef2260208b15ddd390007fdd552?permalink_comment_id=4592230#gistcomment-4592230
@ashishvaishno You cannot use Kube config that requires aws to authenticate. Unfortunately, I don't remember how I created it in the past but I do remember that there is such an option.
@vfarcic I also tried to create a sa and a kubernetes.io/service-account-token, but this also didnt work 😢
I cannot create client certificate as well, cos EKS currently does not support Kubelet client auth
Not sure what else to try 😞
I'm not sure how to do it. I remember that, in the past, I was generating EKS Kube config that does not require aws CLI but I don't remember anymore how I did it and since I moved to Crossplane, I've been using it to generate other clusters so I haven't faced that issue in a long while.
Unfortunately, I don't have time to try it out myself today and then I'll be traveling for 4-5 weeks and probably won't be in front of a computer (except for work). If Google-ing does not work, try creating a small EKS cluster with Crossplane. That will result in a Secret with Kube config. You might be able to reconstruct it from there.
I followed the instructions and clusterclaims is: Synced: true but Ready: false
seeing this:
Warning SelectComposition 47s (x10 over 4m48s) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot select Composition: no compatible Compositions found
Nothing is being deployed, as i would expect, like vpc, subnets, etc and then cluster
@jecausey I made quite a few changes (improvements) in that repo since I created this Gist and one of those broke the demo. I just updated the Gist. It should work now.
@vfarcic Thank you!!! Totally working now!
I'm getting an error at line 68 with fresh microk8s:
No commands reported errors until then.
Any help appreciated.