I've been working on improving operational visibility of a system that uses Kafka as its data backbone that have a combination of size and age-based retention policies [with no compaction]. One of the key metrics I want to track is how much time is actually held in some of these topics, before I start making changes, as a longer amount of time means I have a longer range of recovery if I need to reprocess that data. Although kafka_exporter does give me useful visibility of topic lag, Kafka doesn't provide any tools specifically for reporting on the timestamp of the earliest message, so with the aid of kafka-console-consumer, some basic scripting, Prometheus and Grafana, now I have the visibility I need, and could set up some useful thresholds for alerting.
To use this in a Grafana dashboard, here's a sample Prometheus query you can use as part of a Singlestat (format the Singlestat with a unit of 'seconds')
time() - max(kafka_topic_partition_earliest_time_seconds{topic=~