-
Create the smallest instance available with the following settings:
- Firewall: no inbound traffic
- Allow access to all Cloud APIs: https://cloud.google.com/logging/docs/setup/python#run-gce
- Check
Install Ops Agent for Monitoring and Logging
in the Observability - Ops Agent section
-
SSH into the instance
-
Check system
$ python3 -V
Python 3.11.2
$ uname -a
Linux instance-20240213-230249 6.1.0-17-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
- Install pip and venv
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv
- Install pipx
sudo apt install pipx
pipx ensurepath
source ~/.bashrc
- Install Meltano
pipx install meltano
- Check versions
meltano --version
- Create project
meltano init --force meltano-project
cd meltano-project
-
Create
logging.yaml
in the root of the Meltano project with the contents oflogging.yaml
from this Gist -
Edit
/etc/google-cloud-ops-agent/config.yaml
with the contents ofops_agent.yaml
from this Gist (might requiresudo
): -
Restart the Ops Agent
sudo systemctl restart google-cloud-ops-agent
- Run a command
$ meltano test
=== Testing completed successfully. 0 test(s) successful. 0 test(s) failed. ===
- Check the Logs Explorer for a message like
Environment 'dev' is active
Hey @edgarrmondragon I figured out a workaround for this. I still haven't been able to get
meltano el
ormeltano run
to get GCP logs sent to GCP cloud logging, but if it's running within a docker container it works by adding the below block to/etc/docker/daemon.json
I find it useful to do this on a VM because running such a long job on GCP cloud run costed about 5x more compared to running it on a VM. Cloud run also currently has a timeout before the job is killed. Here are the steps I took:
enable-osconfig
andenable-guest-attributes
to TRUE/etc/docker/daemon.json
git clone <repo>
and set.env
variables / configurations / secretsscreen
sudo docker build -t <my-image> . && sudo docker run --rm -it --env-file=.env --log-driver=gcplogs <my-image>