Skip to content

Instantly share code, notes, and snippets.

@pentium10
Created May 18, 2021 16:34
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pentium10/e9ce523bd0be0c83e3d3cdb12026ace3 to your computer and use it in GitHub Desktop.
Save pentium10/e9ce523bd0be0c83e3d3cdb12026ace3 to your computer and use it in GitHub Desktop.
#workflow entrypoint
main:
steps:
- getList:
call: BQ_Query
args:
query: select distinct _TABLE_SUFFIX as table_id FROM `my-project.analytics_242990349.*`
WHERE _TABLE_SUFFIX
BETWEEN CONCAT('events_intraday_', format_date('%Y%m%d', date_sub(CURRENT_DATE, interval 7 day)))
AND CONCAT('events_intraday_', format_date('%Y%m%d', CURRENT_DATE))
result: items
- loopItems:
call: BQ_Results_LoopItems
args:
items: ${items.rows}
result: res
- final:
return: ${res}
BQ_Query:
params: [query]
steps:
- runBQquery:
call: googleapis.bigquery.v2.jobs.query
args:
projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
body:
useLegacySql: false
query: ${query}
result: queryResult
- documentFound:
return: ${queryResult}
BQ_Results_LoopItems:
params: [items]
steps:
- init:
assign:
- i: 0
- result: ""
- check_condition:
switch:
- condition: ${len(items) > i}
next: iterate
next: exit_loop
- iterate:
steps:
- process_item:
call: BQ_Task
args:
table_id: ${items[i].f[0].v}
result: result
- assign_loop:
assign:
- i: ${i+1}
next: check_condition
- exit_loop:
return: ${result}
BQ_Task:
params: [table_id]
steps:
- delete:
call: BQ_Query
args:
query: ${"DELETE FROM `my-project.analytics_242990349."+table_id+"`
WHERE event_name IN ('not a real event')"}
result: queryResult
- documentFound:
return: ${queryResult}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment