sketching out a how to get logs from heroku to grafana cloud

sketched-ideas.md

Hi there,

I'm looking for an easy way to get data out of heroku, and into Loki (ideally hosted on Grafana Cloud), and I think I need some pointers on how to get data out of Heroku's logs, and then into Loki, with the rest of logs aggregated from other sources.

Are the services or tools for doing this so far?

I started a twitter thread below with my notes so far, but I can't be the only person who would find this useful.

So I figured here might be wise to ask before I try doing so myself.

https://twitter.com/mrchrisadams/status/1349373487585107968

How I think you could do it - 3 ways:

I think there are three main ways, and I'd welcome some pointers before I try implementing them.

1. on a cron job
2. using a promtail instance on heroku
3. using a HTTP API forwarder instance on heroku

the Cronjob approach

One you could conceivably do on a cronjob. You can fetch logs for an app 1500 lines at a time using the CLI with something like this

heroku logs -a your-npp-name -n 1500

See more about accessing heroku logs via the CLI.

If you could persist the timestamp of the last logs, presumably you could send them in batches every N minutes to the Loki push API, using some code that looks a bit like so:

import requests
import json
import datetime
import pytz

host = 'somehost'
curr_datetime = datetime.datetime.now(pytz.timezone(CORRECT_TIMEZONE
curr_datetime = curr_datetime.isoformat('T')
msg = 'On server {host} detected error'.format(host=host)

# push msg log into grafana-loki
url = 'http://host-where-loki-run:3100/api/prom/push'
headers = {
    'Content-type': 'application/json'
}
payload = {
    'streams': [
        {
            'labels': '{source=\"Name-of-your-source\",job=\"name-of-your-job\", host=\"' + host + '\"}',
            'entries': [
                {
                    'ts': curr_datetime,
                    'line': '[WARN] ' + msg
                }
            ]
        }
    ]
}
payload = json.dumps(payload)
answer = requests.post(url, data=payload, headers=headers)
# end pushing

There's all kinds of reasons this isn't ideal, but it might work for quiet services.

the promtail approach

Another option I see would be to run promtail on heroku itself, and have that act as a collector to send logs on to grafana.

You could probably get away with just using a small hobby dyno, as promtail is quite lightweight.

If you're not that confident with Go though, you might not prefer to do so, and it's not obvious to me yet how you'd forward along the tags to identify each app.

the HTTP API forwarder

Finally, and lets assume we're using python as the example above is in that language, you deploy a small python app using something like FastAPI or similar.

You'd then set up a log drain to POST logs to it in batches via HTTP, then have that API push those into Loki using the Push API as I've outlined above.

It's also a fairly simple API to implement. You receive POSTed data, figure out the app that sent it (maybe from the provided log drain token), and then POST almost the same payload to the Loki end point.

Of the three options, I think the third one is the probably the one I'm leaning towards.

Given how popular Heroku is, and how popular Grafana is, I figured this might already be in existence.

Has someone already implemented any of these components?

What did you end up using?