Skip to content

Instantly share code, notes, and snippets.

View andrevrochasilva's full-sized avatar

André Rocha Silva andrevrochasilva

View GitHub Profile
import os
import twitter
from datetime import datetime, timedelta
def hello_cloudscheduler(event, context):
# API keys
consumer_key = os.environ['CONSUMER_KEY']
consumer_secret = os.environ['CONSUMER_SECRET']
@andrevrochasilva
andrevrochasilva / cloudfn.js
Last active December 9, 2023 19:03
Codes used to make the POC with dataflow, cloud functions and BigQuery
const { google } = require('googleapis');
const dataflow = google.dataflow('v1b3');
const TEMPLATE_BUCKET = `your-bucket`;
module.exports.kickOffDataflow = async (event, event_description) => { // Check if the file was put in 'input/' folder
if (!event['id'].includes('/input/')){
return
}
const input = 'gs://' + event['id'].split('/').slice(0, -1).join('/')
const jobName = `citiesonbigquery`;
const tmpLocation = `gs://${TEMPLATE_BUCKET}/tmp`;