Custom logging handlers in Meltano

README.md

Send Meltano logs to Cloud Logging on GCP Compute Engine

1. Create the smallest instance available with the following settings:

   1. Firewall: no inbound traffic
   2. Allow access to all Cloud APIs: https://cloud.google.com/logging/docs/setup/python#run-gce
   3. Check Install Ops Agent for Monitoring and Logging in the Observability - Ops Agent section

2. SSH into the instance

3. Check system

$ python3 -V
Python 3.11.2

$ uname -a
Linux instance-20240213-230249 6.1.0-17-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

4. Install pip and venv

sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv

5. Install pipx

sudo apt install pipx
pipx ensurepath
source ~/.bashrc

6. Install Meltano

pipx install meltano

7. Check versions

meltano --version

8. Create project

meltano init --force meltano-project
cd meltano-project

9. Create logging.yaml in the root of the Meltano project with the contents of logging.yaml from this Gist

10. Edit /etc/google-cloud-ops-agent/config.yaml with the contents of ops_agent.yaml from this Gist (might require sudo):

11. Restart the Ops Agent

sudo systemctl restart google-cloud-ops-agent

14. Run a command

$ meltano test


=== Testing completed successfully. 0 test(s) successful. 0 test(s) failed. ===

15. Check the Logs Explorer for a message like Environment 'dev' is active

References

1. https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/install-agent-vm-creation
2. https://cloud.google.com/logging/docs/agent/ops-agent/configuration

logging.yaml

version: 1
disable_existing_loggers: false

formatters:
  json:
    (): meltano.core.logging.json_formatter
handlers:
  file:
    class: logging.FileHandler
    level: DEBUG
    filename: /tmp/meltano.log
    formatter: json
root:
  level: DEBUG
  handlers: [file]

ops_agent.yaml

logging:
  receivers:
    meltano:
      type: files
      include_paths:
      - /tmp/meltano.log  # Path to the Meltano log file
  processors:
    json:
      type: parse_json
    map_severity:
      type: modify_fields
      fields:
        severity:
          move_from: jsonPayload.level
  service:
    pipelines:
      meltano:
        receivers: [meltano]
        processors: [json, map_severity]

@edgarrmondragon Ah sorry just realized the approach I mentioned logs everything as default severity. Unfortunately I still haven't found a way to track the meltano logs along with the severity level from a local machine or VM -> GCP cloud logging. This approach also only works if using docker and meltano together.

@melgazar9 I've been really busy as of late, but I'd like to come back to this and perhaps create a small terraform recipe for deploying Meltano to GCP CE, while sending logs to GCP cloud logging.