Google Cloud provides a vast array of managed services for computing, databases, NoSQL, and more. While these services offer numerous benefits, it's essential to weigh the tradeoff between cost and advantages.
Managed services are often the better choice as they ensure data security and eliminate the need for additional configuration and maintenance.
But using Google Compute Engine (GCE) to run cronjobs can provide a more cost-effective solution than Google App Engine (GAE) and enable more powerful hardware to process complex tasks in a unmanaged service.
To set up cronjobs on GCE, a well-defined process can be followed, which is documented below.
Execute a cronjob in GCE
- Get data from Cloud SQL
- Feed Datastore
- Use the data in App Engine.
Configure a Service Account with minimun required permissions. In this case we need to set:
- Cloud Datastore User (Datastore/Firebase)
- Cloud SQL Client (Cloud SQL)
- Secret Manager Secret Accessor (Django variables - used in script)
- Secret Manager Viewer (Django variables - used in script)
- Logs Writer (For monitoring, very useful for GCE Observability)
- Monitoring Metric Writer (Same as above)
GCE is a VPS, a non managed server where you boot and manage your own server from scratch.
Apart from setup the user to run the services in GCE, and basic softwares installation we will use, these are the main steps to make the box to work with other GCP services.
Install the public keys of a user allowed to clone and pull the project repo.
Extract data from a Cloud SQL database using GCP cloud_sql_proxy
nohup <path_to>/cloud_sql_proxy -instances="<CONN STRING>"=tcp:5440 &
You will probably need to activate the service account in the environment for cloud_sql_proxy to run in behalf.
gcloud auth activate-service-account --key-file=<SERVICE ACCOUNT KEYFILE PATH>
gcloud auth list # Only to check
Set the variables to run your jobs
# For Python/Django project path
PYTHONPATH=<PROJECT PATH>
# For GCP access
GOOGLE_CLOUD_PROJECT=<GCP PROJECT ID>
GOOGLE_APPLICATION_CREDENTIALS=<SERVICE ACCOUNT KEYFILE PATH>
# Cronjob as usual
# Activate a service account, and execute GCP process in behalf of the account.
<path_to>n/gcloud auth activate-service-account --key-file=<SERVICE ACCOUNT KEYFILE PATH> && source <VIRTUALENV> && <VENV PATH TO>/python <PYTHON JOB>