-
Create the smallest instance available with the following settings:
- Firewall: no inbound traffic
- Allow access to all Cloud APIs: https://cloud.google.com/logging/docs/setup/python#run-gce
- Check
Install Ops Agent for Monitoring and Logging
in the Observability - Ops Agent section
-
SSH into the instance
-
Check system
$ python3 -V
Python 3.11.2
$ uname -a
Linux instance-20240213-230249 6.1.0-17-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
- Install pip and venv
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv
- Install pipx
sudo apt install pipx
pipx ensurepath
source ~/.bashrc
- Install Meltano
pipx install meltano
- Check versions
meltano --version
- Create project
meltano init --force meltano-project
cd meltano-project
-
Create
logging.yaml
in the root of the Meltano project with the contents oflogging.yaml
from this Gist -
Edit
/etc/google-cloud-ops-agent/config.yaml
with the contents ofops_agent.yaml
from this Gist (might requiresudo
): -
Restart the Ops Agent
sudo systemctl restart google-cloud-ops-agent
- Run a command
$ meltano test
=== Testing completed successfully. 0 test(s) successful. 0 test(s) failed. ===
- Check the Logs Explorer for a message like
Environment 'dev' is active
Hey unfortunately nope I haven't. The closest I could get is
sudo docker-compose up | logger 2>&1
(ormeltano el <tap> <target> | logger 2>&1
) but that doesn't capture the severity level. I'm going to try and use cloud run (or something similar). It seems like their serverless solutions gravitate towards best practices, but not totally sure. Cloud run has a 60m timeout, so that might be out of the question for long-running ELT processes.