prometheus-auth fix token

gistfile1.txt

# This behaviour comes in Kubernetes 1.21 with a change of Serviceaccount token format. Bound tokens become default, legacy
# token still alive. In prometheus-auth, a middleware
# for bearer token auth in front of prometheus, a SubjectAccessReview will looking for a token, provided by the federated
# prometheus instance via http endpoint. This token is searched in a list of secrets cluster-wide and if there is a match, 
# comparing process started for ServiceAccount and Namespace. After success, a hijack federate will provide access to the
# related cluster metrics. Normaly, the token in the federate call and the content of the related secret of the 
# ServiceAccount is equal. In newer Kubernetes version this format is rewritten and internally the new format of bound token 
# is stored.
ä With this difference, the SubjectAccessReview is denied and no metrics are accessable. Unfortunatelly the related
# ServiceMonitor for federate is UP, no errors are visiable.
# howto fix:
kubectl -n cattle-prometheus edit prometheus cluster-monitoring
# replace version with a debug output of broken namespaces
# image: mtr.devops.telekom.de/rancher/prometheus-auth:v0.3.0
# with
# image: mtr.devops.telekom.de/eumel8/prometheus-auth:master
#
# watch for broken namespaces
kubectl -n cattle-prometheus logs prometheus-cluster-monitoring-0 -c prometheus-agent -f
INFO[2023-04-19T20:17:55Z] listening on 10.42.17.86:9090, proxying to http://127.0.0.1:9090 with ignoring 'remote reader' labels [prometheus,prometheus_replica], only allow maximum 512 connections with 5m0s read timeout . 
INFO[2023-04-19T20:17:55Z] Start listening for connections on 10.42.17.86:9090 
WARN[2023-04-19T20:19:10Z] missing namespace token: cattle-prometheus-p-w9rrg 
# extract token from prometheus pod
kubectl -n cattle-prometheus-p-rggct exec -it prometheus-project-monitoring-0 -c prometheus -- cat /var/run/secrets/kubernetes.io/serviceaccount/token  | base64
# replace token value with the previous output
kubectl -n cattle-prometheus-p-rggct edit secret project-monitoring-token-p7dtr
# try this together
kubectl -n cattle-prometheus-p-rggct patch secret project-monitoring-token-p7dtr  --patch="{\"data\": { \"token\": \"$(kubectl -n cattle-prometheus-p-rggct exec -it prometheus-project-monitoring-0 -c prometheus -- cat /var/run/secrets/kubernetes.io/serviceaccount/token  | base64)\" }}"
# query project prometheus with `kube_pod_container_info`

# ref 
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1205-bound-service-account-tokens
# https://github.com/rancher/prometheus-auth/blob/master/pkg/kube/namespaces.go
# https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-bound-service-account-tokens?hl=en

remark: This workaround has only a live time around 1 hour, until the validation date of the Bound Token changed.
The program logic must be completely changed. This is done in rancher/prometheus-auth#16