Conjur K8s Authenticator Debugging

debugging_info.md

• Display role bindings for conjur-cluster service account token

  oc get clusterrolebindings -o json \
    | jq '.items | map(select(any(.subjects[]; .name | contains("conjur-cluster"))))'
  

• Display conjur-authenticator role information

  oc describe clusterrole conjur-authenticator
  

• Display configured K8s CA certificate

  conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/ca-cert
  

• Verify service account token is what's expected

  This does not output the service token itself, but does display the MD5 sum of the token.

  TOKEN_SECRET_NAME="$(kubectl get secrets -n <FOLLOWER_NAMESPACE> \
      | grep 'conjur.*service-account-token' \
      | head -n1 \
      | awk '{print $1}')"
  
  # Show MD5 sum for expected token
  oc get secret -n <FOLLOWER_NAMESPACE> $TOKEN_SECRET_NAME -o json \
      | jq -r .data.token \
      | base64 --decode \
      | md5sum
  
  # Show MD5 sum for stored token
  conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/service-account-token \
      | md5sum
  

• Display configured API URL for authenticator

  echo "$(conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/api-url)"
  

• Display generated CA certificate for authenticator

  conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/ca/cert
  

• Retrieve metadata about generated CA key:

  Conjur show demo:variable:conjur/authn-k8s/<AUTHENTICATOR_ID>/ca/key
  

• List configured authenticators:

  curl -ks https://localhost/info | jq .authenticators
  

@micahlee
Can we add some common error messages to this as well?

Log from POD:
Starting follower services...
Joined session keyring: 233229515
*** Killing all processes...
Traceback (most recent call last):
File "/sbin/my_init", line 414, in
/tmp/seedfile/start-follower.sh: line 6: 10 Terminated sleep 5
main(args)
File "/sbin/my_init", line 330, in main
import_envvars(False, False)
File "/sbin/my_init", line 90, in import_envvars

If the above is encountered, the service account needs to be allowed to run as root. To correct, run: oc adm policy add-scc-to-user anyuid "system:serviceaccount:$CONJUR_NAMESPACE_NAME:$CONJUR_SERVICEACCOUNT_NAME"

Another one:

ERROR: 2019/10/03 22:25:34 main.go:53: Failure authenticating: client certificate not found at /etc/conjur/ssl/client.pem

The authenticator doesn't have sufficient privileges through the OCP/K8s API to create the folder/file in the init container for the client certificate. Either create the folder in the container manually, or use the updated seed-fetcher image from Docker Hub.

Nathan Whipple <notifications@github.com> writes:

@micah

wrong micah :)

…

-- micah

@whip113, these are good additions, thanks! I will add these to the Confluence doc we have for the appliance troubleshooting.

Script to verify conjur service account access to K8s API using service-account token, cert & API URL:

#!/bin/bash

CLI=kubectl
CONJUR_NAMESPACE_NAME=cyberark

# Use a cap-D for decoding on Macs
if [[ "$(uname -s)" == "Linux" ]]; then
  BASE64D="base64 -d"
else
  BASE64D="base64 -D"
fi

TOKEN_SECRET_NAME="$($CLI get secrets -n $CONJUR_NAMESPACE_NAME \
    | grep 'conjur.*service-account-token' \
    | head -n1 \
    | awk '{print $1}')"
CERT="$($CLI get secret -n $CONJUR_NAMESPACE_NAME $TOKEN_SECRET_NAME -o json \
      | jq -r '.data["ca.crt"]' \
      | $BASE64D)"
TOKEN="$($CLI get secret -n $CONJUR_NAMESPACE_NAME $TOKEN_SECRET_NAME -o json \
      | jq -r .data.token \
      | $BASE64D)"
API="$($CLI config view --minify -o yaml | grep server | awk '{print $2}')"

echo "$CERT" > k8s.crt
if [[ "$(curl -s --cacert k8s.crt --header "Authorization: Bearer ${TOKEN}" $API/healthz)" == "ok" ]]; then
  echo "Service account access to K8s API verified."
else
  echo
  echo ">>> Service account access to K8s API NOT verified. <<<"
  echo
fi
rm k8s.crt

Thanks for that script @jodyhuntatx ! and for this page all :)

@micahlee, can we get examples of what good values look like for the variable values we're checking? I've seen in the past where there are double quotes or blank spaces.

Client authentication failure. Error in master/leader log was:

CONJ00048I Authentication Error: #<Errors::Authentication::AuthnK8s::MissingClientCertificate: CONJ00029E Client SSL certificate is missing from the header>

The client log showed it was receiving the certificate (login succeeded), but then failing when trying to use the cert for authentication.

DEBUG: 2022/02/24 16:42:55.132874 authenticator.go:267: CAKC039 Trying to log in to Conjur...
DEBUG: 2022/02/24 16:42:55.132881 authenticator.go:158: CAKC041 Logging in as user 'host/test-app'
DEBUG: 2022/02/24 16:42:55.133460 requests.go:22: CAKC045 Login request to: https://t-cdap-follower-svc.test.hcscint.net/authn-k8s/test-cluster/inject_client_cert
DEBUG: 2022/02/24 16:42:55.557182 file.go:54: CAKC051 Waiting for file /etc/conjur/ssl/client.pem to become available...
DEBUG: 2022/02/24 16:42:55.607547 file.go:54: CAKC051 Waiting for file /etc/conjur/ssl/client.pem to become available...
DEBUG: 2022/02/24 16:42:55.607667 authenticator.go:224: CAKC049 Loaded client certificate successfully from /etc/conjur/ssl/client.pem
DEBUG: 2022/02/24 16:42:55.607869 authenticator.go:241: CAKC050 Deleted client certificate from memory
DEBUG: 2022/02/24 16:42:55.607886 authenticator.go:273: CAKC036 Logged in
DEBUG: 2022/02/24 16:42:55.607908 authenticator.go:256: CAKC042 Cert expires: 2022-02-27 16:42:55 +0000 UTC
DEBUG: 2022/02/24 16:42:55.607916 authenticator.go:257: CAKC043 Current date: 2022-02-24 16:42:55.607894465 +0000 UTC
DEBUG: 2022/02/24 16:42:55.607924 authenticator.go:258: CAKC044 Buffer time: 30s
DEBUG: 2022/02/24 16:42:55.608595 requests.go:46: CAKC046 Authn request to: https://t-cdap-follower-svc.test.hcscint.net/authn-k8s/test-cluster/Tst1/host%2Ftest-app/authenticate
DEBUG: 2022/02/24 16:42:55.608621 authenticator.go:318: CAKC069 Sending authn-k8s request...
ERROR: 2022/02/24 16:42:55.658420 main.go:49: CAKC016 Failed to authenticate

Clearly the cert was not reaching the Follower. Turns out the load balancer was terminating TLS and stripping the cert out of the header before sending to the Follower. Configuring the LB for pass-through allowed authentication to succeed.