Short Term Epic: Set Up Cloud Composer and Migrate Vertica Data to BigQuery
- NetOps Setup for GCP and Provisioning
- Goal: Establish a network setup in GCP that integrates with on-prem infrastructure and provision Cloud Composer using Terraform.
- Key Steps:
- Create a /18 subnet for scalability.
- Provision Cloud Composer clusters for Test, Staging, and Production.
- Ensure clusters are located close to BigQuery to minimize networking costs.
- Set up local development with Minikube to mirror Cloud Composer environments.
- Container and Registry Setup