Skip to content

Instantly share code, notes, and snippets.

@polleyg
Last active June 22, 2018 17:48
Show Gist options
  • Save polleyg/91b5ab6bdc2b8d1104ae2b22f2b30eb2 to your computer and use it in GitHub Desktop.
Save polleyg/91b5ab6bdc2b8d1104ae2b22f2b30eb2 to your computer and use it in GitHub Desktop.
How to coalesce Stackdriver logs/sinks into just one BigQuery project/dataset

Exporting Stackdriver logs back to BigQuery for analysis is a very straightforward task (and a very good pattern to follow too BTW). However, when you are managing lots of projects you'll probably want to coalesce the logs from all your projects into one dedicated BigQuery project/dataset for easier querying and maintainability.

However, setting this up in the GCP console isn't that obvious to most users. This is because the configuration in the Stackdriver sink config doesn't actually allow you to select a different project/dataset to send the logs to. It only lets you select the currently selected/working project:

oppmi

The tip/trick is to select Custom destination from the drop-down and then use the following format to specify the project/dataset where you want the logs to go. Plug in the project and dataset that you want to use:

bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]

After you've done that, all you need to do is to give the sink writer editor permissions on the said project/dataset, and you're done.

Easy-peasy-lemon-squeezy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment