Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to register Rancher managed Kubernetes clusters in Argo CD

How to register Rancher managed Kubernetes clusters in Argo CD

Registering Rancher managed clusters in Argo CD doesn't work out of the box unless the Authorized Cluster Endpoint is used. Many users will prefer an integration of Argo CD via the central Rancher authentication proxy (which shares the network endpoint of the Rancher API/GUI). So let's find out why registering clusters via Rancher auth proxy fails and how to make it work.

Hint: If you are just looking for the solution scroll to the bottom of this page.

Why do i get an error when running argocd cluster add?

Service Account tokens and the Rancher authentication proxy

Registering external clusters to an Argo CD instance is normally accomplished by invoking the command-line tool argocd like this:

$ argocd cluster add <context-name>

Here, context-name references a context in the command-line user's kubeconfig file (by default ~/.kube/config).
Running this command using a context that points to a Rancher authentication proxy endpoint (typically an URL in the form https://<rancher-server-endpoint>/k8s/clusters/<cluster-id>) will result in the following error:

FATA[0001] rpc error: code = Unknown desc = REST config invalid: the server has asked for the client to provide credentials 

Ref: GH ticket

Before getting to the solution let's understand why this happens.

By default, the kubeconfig files provided by Rancher specify the Rancher server network endpoint as the cluster API server endpoint. By doing so Rancher acts as an authentication proxy that validates the user identity and then proxies the request to the downstream cluster.

This generally has a number of advantages compared to having clients communicating directly with the downstream cluster API endpoint:One of the is that this provides a high available Kubernetes API endpoint for all clusters under Rancher's management, sparing the ops team from having to maintain a failover/load-balancing mechanism for each cluster's API servers.
An alternative authentication method avalaible in Rancher is the Authorized Cluster Endpoint which allows requests to be authenticated directly at the downstream cluster Kubernetes API server. See the documentation for details on these methods.

It is important to understand that only the Authorized Cluster Endpoint allows authentication based on K8s service account tokens. The authentication proxy endpoint requires usng a Rancher API token instead.

How can still use the central Rancher auth endpoint to integrate ArgoCD

To summarize: In order to integrate Argo CD via the Rancher server network endpoint, we will need to setup Argo CD with a Rancher API token in lieu of a Kubernetes Service Account token.

For now this can not be accomplished using the argocd command-line tool because it doesn't let the user specify a pre-existing API credential or custom kubeconfig.

Luckily, argocd is at the core just a Kubernetes client with syntactic sugar coating and therefore does most things by interacting with Kubernetes (CRD) resources under the hood. So whatever $ argocd cluster add does, we should be able to do this using kubectl and K8s manifests.

According to the documentation the "cluster add" command does the following:

The above command installs a ServiceAccount (argocd-manager), into the kube-system namespace of that kubectl context, and binds the service account to an admin-level ClusterRole. Argo CD uses this service account token to perform its management tasks (i.e. deploy/monitoring).

And in another place we find:

To manage external clusters, Argo CD stores the credentials of the external cluster as a Kubernetes Secret in the argocd namespace. This secret contains the K8s API bearer token associated with the argocd-manager ServiceAccount created during argocd cluster add, along with connection options to that API server

Futhermore, the format of that cluster secret is also described in detail on this page providing the following example:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster.com
  server: https://mycluster.com
  config: |
    {
      "bearerToken": "<authentication token>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 encoded certificate>"
      }
    }

Finally, the argocd CLI creates a RBAC role called argocd-manager-role which by default assigns clusteradmin privileges but can be narrowed down to implement a least-privilege concept.

tl;dr: Here are the steps to register Rancher managed clusters using the central Rancher API endpoint

So it appears that all we need to do is replace the argocd cluster add command with the following steps:

  1. Create a local Rancher user account (e.g. service-argo)
  2. Create a Rancher API token for that user account, either by logging in and using the GUI (API & Keys -> Add Key) or requesting the token via direct invocation of the /v3/tokens API resource.
  3. Authorize that user account in the cluster (GUI: Cluster -> Members -> Add) and assign the cluster-member role (role should be narrowed down for production usage later).
  4. Create a secret resource file (e.g. cluster-secret.yaml) based on the example above, providing a configuration reflecting the Rancher setup:
    • name: A named reference for the cluster, e.g. "prod".
    • server: The Rancher auth proxy endpoint for the cluster in the format: https://<rancher-server-endpoint>/k8s/clusters/<cluster-id>
    • config.bearerToken: The Rancher API token created above
    • config.tlsClientConfig.caData: PEM encoded CA certificate data for Rancher's SSL endpoint. Only needed if the server certificate is not signed by a public trusted CA.
  5. Then apply the secret to the Argo CD namespace in the cluster where Argo CD is installed (by default argocd): $ kubectl apply -n argocd -f cluster-secret.yaml

Finally check that the cluster has been successfully registered in Argo CD:

argocd cluster list
SERVER                                                     NAME     VERSION  STATUS      MESSAGE
https://xxx.rancher.xxx/k8s/clusters/c-br1xm               vsphere  1.17     Successful
https://kubernetes.default.svc                                               Successful
@papanito

This comment has been minimized.

Copy link

@papanito papanito commented Jun 11, 2020

Create a local Rancher user account (e.g. service-argo)

So this is just [x] Standard User without "Custom" permissions

@papanito

This comment has been minimized.

Copy link

@papanito papanito commented Jun 11, 2020

It works!! Thanks @janeczku

@papanito

This comment has been minimized.

Copy link

@papanito papanito commented Jun 11, 2020

@janeczku darf man das sharen?

@Krstffer

This comment has been minimized.

Copy link

@Krstffer Krstffer commented Jun 11, 2020

Great solution, really helpful!

@janeczku

This comment has been minimized.

Copy link
Owner Author

@janeczku janeczku commented Jun 11, 2020

@papanito Klar kann man das sharen. Habe mich mit den Berechtigungen nicht näher beschäftigt, denke aber dass vermutlich sogar die minimale, globale Rolle User Base ausreicht.
Screenshot 2020-06-11 at 11 41 01

@papanito

This comment has been minimized.

Copy link

@papanito papanito commented Jun 11, 2020

@janeczku danke für die info und nochmals herzlichen Dank für die Lösung. @rancher rocks! 🎉

@stealthcold

This comment has been minimized.

Copy link

@stealthcold stealthcold commented Jul 22, 2020

It was very helpful! Thanks!

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Aug 20, 2020

Just one note on the documentation on the secret resource definition: cluster-secret.yaml. The server: and config.tlsClientConfig.caData: values can be found in the downstream Rancher cluster kubeconfig file. I initially failed to get it to work by using the Rancher console endpoint URL and using the public key for the SSL certificate of the Rancher HA cluster. I had to reread the documentation terminology carefully to make sure that I was sourcing the correct key:value pairs.

Thank you for creating this document!

@andrewcottle

This comment has been minimized.

Copy link

@andrewcottle andrewcottle commented Sep 17, 2020

I am not having any luck, unfortunately. thank you for the writeup.

  • Is the local user account supposed to be created for the cluster that needs to be added? I assumed yes.

  • The server I assume is supposed to be the endpoint for the cluster you are adding

  • Is the bearerToken supposed to be just the "secret key" portion? When its created in rancher, it provides the access key (username), secret key (password) and then the Bearer Token (access key:secret key). Does it need to be encoded or just as it is given by rancher?

  • The caData portion, one user mentioned it can be retrieved from the kubeconfig file. Is it the "certificate-authority-data" value or is it the token value from the users section of kubeconfig? Or something else? If its the token from the users section does it need to be base64 encoded? I assume yes. For instance the token is in this format in my kubeconfig (for the user of the cluster that is to be added)

users:
- name: dev-rke
user:
token: kubeconfig-user-6kfddw.b-894fd:5439jgf03jg9030vm30yj3930mv3jm3v9m39b

Should it be that entire string? Or only the part after the colon?

I know i asked a lot, but I am just trying to get this second cluster added and i have already wasted a whole day on this. Any help appreciated.

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Sep 17, 2020

@andrewcottle

  1. The local user needs standard user global permissions and Cluster Member permissions on the cluster where ArgoCD is installed and on the clusters that it will deploy applications
  2. The server value is found in the endpoint cluster ~/.kube/config which you can get from the Rancher console
  3. The bearer token you get by logging in as the local user account and creating the API key. You don't change the value.
  4. The caData is the certificate-authority-data from the endpoint cluster ~/.kube/config which you can get from the Rancher console. Copy and paste the quoted key value. It is under the cluster section of the ~/.kube/config. Don't use the user: token: value.

I had similar issues. You have to pay close attention to the terminology that he uses to configure the secret CRD properly. If you try to use the ~/.kube/config user: token: value it won't work.

@andrewcottle

This comment has been minimized.

Copy link

@andrewcottle andrewcottle commented Sep 17, 2020

@mkolb-navican

Thank you so much! The main thing I was missing was that the local user needed to be a member of both clusters! I tried both the certificate-authority-data and token from kubeconfig. But adding that user as a member of the other cluster did the trick.
Much appreciated.

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Sep 17, 2020

@andrewcottle You are welcome. I am just paying it forward for all the others that have helped me. Take care and stay safe!

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Oct 6, 2020

I am only able to get the https://kubernetes.default.svc in-cluster to work. It seems like the default install wants to add the in-cluster and https://<rancher-server-endpoint>/k8s/clusters/<cluster-id> for the cluster where ArgoCD is installed. I setup the secret, but I still get the credential error trying to add the local cluster https://<rancher-server-endpoint>/k8s/clusters/<cluster-id>.

@andrewcottle

This comment has been minimized.

Copy link

@andrewcottle andrewcottle commented Oct 7, 2020

@mkolb-navican so you arent able to add additional clusters managed by Rancher to argocd?

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Oct 8, 2020

Yes, in-cluster works, but adding another cluster doesn't. I created two bearer tokens of cluster scope for a service-argocd local account. One for each cluster. The error events are definitely user permissions for the second cluster. I even get errors for the local cluster that is added at the same time as the in-cluster resource.

@andrewcottle

This comment has been minimized.

Copy link

@andrewcottle andrewcottle commented Oct 8, 2020

@mkolb-navican for me I didnt have to do anything to get in-cluster, e.g. the cluster where argocd is running to work. To get the second Rancher Cluster to be managed correctly by argocd I did the following:

  • Create a local Rancher user account (e.g. service-argo)

  • Login to rancher using the new user, create a Rancher API token for the new user account, either by logging in and using the GUI (API & Keys -> Add Key) or requesting the token via direct invocation of the /v3/tokens API resource.
    -- The Rancher API token (Bearer) in format: token-abcded:abcde12345abcde12345abcde1234abcde1234abcde12345

  • Authorize that user account in ALL target clusters (GUI: Cluster -> Members -> Add) and assign the cluster-member role in ALL target clusters
    -- The local user needs standard user global permissions and Cluster Member permissions on the cluster where ArgoCD is installed and on the clusters that it will deploy applications

  • Create a secret resource file (e.g. cluster-secret.yaml) based on the example above, providing a configuration reflecting the Rancher setup:

kind: Secret
metadata:
  name: dev-rke
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: dev-rke
  server: https://<rancher-server-endpoint>/k8s/clusters/<cluster-id>
  config: |
    {
      "bearerToken": "<The Rancher API token created above>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<caData is the certificate-authority-data from the endpoint cluster>"
      }
    }
  • Then apply the secret to the Argo CD namespace in the cluster where Argo CD is installed (by default argocd): $ kubectl apply -n argocd -f cluster-secret.yaml
@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Oct 8, 2020

@andrewcottle Did you select the API key scope as "no scope" vs. a specific cluster scope? If so, this would explain why my permissions are failing.

@andrewcottle

This comment has been minimized.

Copy link

@andrewcottle andrewcottle commented Oct 8, 2020

No scope.

@mkolb-navican

This comment has been minimized.

Copy link

@mkolb-navican mkolb-navican commented Oct 8, 2020

I am not able to get this to work... Does anyone else have Rancher in an HA environment and able to add additional downstream clusters?

@ansilh

This comment has been minimized.

Copy link

@ansilh ansilh commented Oct 16, 2020

Clone "Cluster Member" role and create a new one with below additional rule and then assign this role to the Rancher user.

image
(Caution: User will be able to "see" all Projects with this rule)
Add namespace list in the secret if you want to restrict ArgoCD to specific namespaces

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: downstream1
  server: https://xxxx/k8s/clusters/c-xxxx
  namespaces: namespace1,namespace2
  config: |
    {
      "bearerToken": "token-xxx:xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "tlsClientConfig": {
        "insecure": false
      }
    }  

You may try to exclude unwanted resources using the ArgoCD's resource exclusion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.