Step-by-step deployment guide in GCP.

gcp_etl_instructions.txt

1. Create a new project in Google Cloud Console
    └── Go to https://console.cloud.google.com/
    └── Click on the project dropdown (top bar) → "New Project"
    └── Give it a name and create it

2. Set up a Cloud SQL Instance (MySQL)
    └── In the search bar, type “Cloud SQL” and open the SQL dashboard
    └── Click “Create Instance” → Choose “MySQL”
    └── Set root password and choose region/zone
    └── Once created, note the **public IP address** for connection

3. Enable Cloud SQL Admin API (if not already enabled)
    └── Navigate to APIs & Services → Library
    └── Search for "Cloud SQL Admin API" and enable it

4. Connect SQL Workbench to the Cloud SQL Instance
    └── Open SQL Workbench and go to “Manage Connections”
    └── Choose MySQL as the DB type
    └── Enter the public IP, root username, and password you set
    └── Test the connection and save it

5. Create an empty database
    └── Use SQL Workbench to run `CREATE DATABASE`
    └── Add foreign keys, primary keys, and data types as per schema design

6. Write the ETL logic in Python
    └── Use `requests`, `BeautifulSoup`, `pandas`, and `sqlalchemy`
    └── Extract:
        └── Wikipedia for city & population
        └── OpenWeatherMap API for weather
        └── AeroDataBox API for flight data
    └── Transform:
        └── Clean, format, and structure data into pandas DataFrames
    └── Load:
        └── Use SQLAlchemy to tranform data into dataframes and export it as Cloud SQL tables

7. Deploy the ETL code as a Google Cloud Function
    └── Go to "Cloud Functions" in GCP
    └── Click “Create Function” → Choose HTTP trigger
    └── Upload your Python script (main.py) and dependencies (requirements.txt)
    └── Set environment variables if needed (like API keys)
    └── Deploy and test

8. Automate the function with Cloud Scheduler
    └── Go to “Cloud Scheduler” and click “Create Job”
    └── Set frequency using cron syntax (e.g., every 12 hours: `0 */12 * * *`)
    └── Select the target to be an HTTP endpoint
    └── Link it to your deployed Cloud Function
    └── Save and enable the job
    └── [Insert Screenshot: Scheduler settings]

This pipeline fetches updated city, weather, and flight data twice a day and 
inserts new data into the appropriate MySQL tables, ensuring everything stays 
up to date with zero manual intervention.