The Prometheus operator offers a simple method to scrape metrics from any Pod. However in many cases the Pod itself is not what you are monitoring but the Pod is used to expose metrics that relate to the Node. In such cases what the user cares about is the Node on which the Pod runs and not the Pod itself.
By default when using PodMonitor all the time series data will have the instance set to the Pod's name. Also the Pod or the Daemon set that the Pod was part of may be deleted, redeployed multiple times over the lifetime of the node. This means that the user will need to perform the mapping between the Pod and the Node on which it run.
However Prometheus allows the instance name (among other labels) to be relabeled in a very simple manner as shown below.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: mycollector
labels:
k8s-app: mycollector
type: mytype
spec:
selector:
matchLabels:
k8s-app: mycollector
namespaceSelector:
matchNames:
- mynamespace
podMetricsEndpoints:
- port: exporter
interval: 10s
relabelings:
- action: replace
sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: instance
Here we specify that original label instance be replaced by __meta_kubernetes_pod_node_name. __meta_kubernetes_pod_node_name is the name of the node the pod is scheduled onto. Many such built in meta data objects are available in prometheus, which can help you modify appropriate labels to make them more meaningful or invariant. The full list of such meta data is available in Prometheus documentation.