Skip to content

Instantly share code, notes, and snippets.

@flaf
Created November 2, 2023 18:36
Show Gist options
  • Save flaf/a5226ae421f4c94b5a29de472825f6ad to your computer and use it in GitHub Desktop.
Save flaf/a5226ae421f4c94b5a29de472825f6ad to your computer and use it in GitHub Desktop.
Why values are added with a range query?

I have a dummy Prometheus exporter which handles the metric test_metric_integer_i with this very basic behavior:

# 1. Before 18:04 the value is always 42. Here the output of /metrics:
test_metric_integer_i{host="telegraf-container"} 42 <a-timestamp-before-18:04>

# 2. Between 18:04 and 18:08, no value at all. The output of /metrics is completely empty.

# 3. After 18:08, the value is always 1000. Here is the output of /metrics:
test_metric_integer_i{host="telegraf-container"} 1000 <a-timestamp-after-18:08>

Now I request the Prometheus API (localhost:9090 in my environment) like this.

First, I try this instant query:

# I have just written a basic shell script to print nicely the values and the date.
~$ curl -sG -X GET http://localhost:9090/api/v1/query \
    --data-urlencode 'query=test_metric_integer_i[5m]' -d 'time=1698944900' \
    | jq '.data.result[0].values' | ./array2pretty.sh
value=42    date=18:03:21
value=42    date=18:03:30
value=42    date=18:03:41
value=42    date=18:03:51 # No value between 18:04 and 18:08 
value=1000  date=18:08:02 #          as expected.
value=1000  date=18:08:11
value=1000  date=18:08:20

To me, this result is what I expected.

And now the range query with the same range as above, ie from "@1698944900-5m" to "@1698944900":

~$ curl -sG -X GET http://localhost:9090/api/v1/query_range \
    --data-urlencode 'query=test_metric_integer_i' \
    -d "start=$((1698944900 - 5*60))" -d 'end=1698944900' -d "step=10s" \
    | jq -r '.data.result[0].values' | ./array2pretty.sh 
value=42    date=18:03:20
value=42    date=18:03:30
value=42    date=18:03:40
value=42    date=18:03:50
value=42    date=18:04:00
value=42    date=18:04:10
value=42    date=18:04:20
value=42    date=18:04:30
value=42    date=18:04:40
value=42    date=18:04:50
value=42    date=18:05:00
value=42    date=18:05:10
value=42    date=18:05:20
value=42    date=18:05:30
value=42    date=18:05:40
value=42    date=18:05:50
value=42    date=18:06:00
value=42    date=18:06:10
value=42    date=18:06:20
value=42    date=18:06:30
value=42    date=18:06:40
value=42    date=18:06:50
value=42    date=18:07:00
value=42    date=18:07:10
value=42    date=18:07:20
value=42    date=18:07:30
value=42    date=18:07:40
value=42    date=18:07:50
value=42    date=18:08:00
value=1000  date=18:08:10
value=1000  date=18:08:20
  1. This is not the resultat I expected. Why this query tells me that the metric has values between 18:04 and 18:08? I think this is a result that makes me think there's something I haven't quite understood about Prometheus. I have a (probably wrong) vision of Prometheus which I compare with influxdb (that I known better) where the values are not sent with a influxQL query. But I think my comparison with influxdb is probably wrong. I'm very interested in explanations and/or links to documentation that explain this behavior.

  2. What kind of range query would reflect the fact that I have no value between 18:04 and 18:08? I have found this request below which seems to me better even if the result is not exactly the same result of the instant query:

~$ curl -sG -X GET http://localhost:9090/api/v1/query_range \
    --data-urlencode 'query=max_over_time(test_metric_integer_i[10s])' \
    -d "start=$((1698944900 - 5*60))" -d 'end=1698944900' -d "step=10s" \
    | jq -r '.data.result[0].values' | ./array2pretty.sh 
value=42    date=18:03:20
value=42    date=18:03:30
value=42    date=18:03:40
value=42    date=18:03:50
value=42    date=18:04:00
value=1000  date=18:08:10
value=1000  date=18:08:20

Thanks in advance for your help.

@flaf
Copy link
Author

flaf commented Nov 3, 2023

It's probably because of the flag query.lookback-delta...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment