#airflow

flow.md

DD-DataDictionary

• description: monitor the AWS RDBMS schema change

• create new column to Summary or Detail:

  1. add COL into google sheet (fill default value it can decrease amount of updating time)
  2. orginzie sql in Summary or Detail func.(fetch table schema from inforamtion_schema table)
  3. col = a.columns[:NUM] in check_cell func -> NUM+=1

• optimation: CDC replacement?

  • too slow to write?
  • can not handle API usuage limt well

• Bottleneck: sleep time update_cell API

• solution

• rate limit

• tenacity