Skip to content

Instantly share code, notes, and snippets.

View aosterloh's full-sized avatar

aosterloh

  • Google
  • Germany
View GitHub Profile
gcs2bq_avro_with_schema = GoogleCloudStorageToBigQueryOperator(
task_id='gcs2bq_avro_with_schema',
bucket='{{var.value.gcs_bucket}}',
source_objects=[
'path/{{ ds_nodash }}/part-*'
],
destination_project_dataset_table='{{var.value.gcq_tempset}}.avro_with_schema{{ ds_nodash }}',
source_format='AVRO',
schema_fields=schemas.gsob(),
create_disposition='CREATE_IF_NEEDED',