Skip to content

Instantly share code, notes, and snippets.

@IslamAzab
Last active July 31, 2019 19:38
Show Gist options
  • Save IslamAzab/9da6d9b4450f9e3db541e6fa3050bb0a to your computer and use it in GitHub Desktop.
Save IslamAzab/9da6d9b4450f9e3db541e6fa3050bb0a to your computer and use it in GitHub Desktop.
when updating BigQuery table, schema changes are always shown

1- Create a new google_bigquery_table

Terraform will perform the following actions:

  # google_bigquery_dataset.test_us_bigquery_dataset will be created
  + resource "google_bigquery_dataset" "test_us_bigquery_dataset" {
      + creation_time      = (known after apply)
      + dataset_id         = "test_schema_update_dataset"
      + etag               = (known after apply)
      + id                 = (known after apply)
      + last_modified_time = (known after apply)
      + location           = "US"
      + project            = (known after apply)
      + self_link          = (known after apply)

      + access {
          + domain         = (known after apply)
          + group_by_email = (known after apply)
          + role           = (known after apply)
          + special_group  = (known after apply)
          + user_by_email  = (known after apply)

          + view {
              + dataset_id = (known after apply)
              + project_id = (known after apply)
              + table_id   = (known after apply)
            }
        }
    }

  # google_bigquery_table.test_us_bigquery_table will be created
  + resource "google_bigquery_table" "test_us_bigquery_table" {
      + creation_time       = (known after apply)
      + dataset_id          = "test_schema_update_dataset"
      + etag                = (known after apply)
      + expiration_time     = (known after apply)
      + id                  = (known after apply)
      + last_modified_time  = (known after apply)
      + location            = (known after apply)
      + num_bytes           = (known after apply)
      + num_long_term_bytes = (known after apply)
      + num_rows            = (known after apply)
      + project             = (known after apply)
      + schema              = jsonencode(
            [
              + {
                  + mode = "REQUIRED"
                  + name = "column_name"
                  + type = "STRING"
                },
              + {
                  + mode = "REQUIRED"
                  + name = "timestamp"
                  + type = "TIMESTAMP"
                },
            ]
        )
      + self_link           = (known after apply)
      + table_id            = "test_schema_update_table"
      + type                = (known after apply)
    }

2- Update table by adding a new column to the schema

Terraform will perform the following actions:

  # google_bigquery_table.test_us_bigquery_table will be updated in-place
  ~ resource "google_bigquery_table" "test_us_bigquery_table" {
        creation_time       = 123
        dataset_id          = "test_schema_update_dataset"
        etag                = "123"
        expiration_time     = 0
        id                  = "project-name:test_schema_update_dataset.test_schema_update_table"
        labels              = {}
        last_modified_time  = 123
        location            = "US"
        num_bytes           = 0
        num_long_term_bytes = 0
        num_rows            = 0
        project             = "project-name"
      ~ schema              = jsonencode(
          ~ [
                {
                    mode = "REQUIRED"
                    name = "column_name"
                    type = "STRING"
                },
              + {
                  + name = "new_column"
                  + type = "FLOAT"
                },
                {
                    mode = "REQUIRED"
                    name = "timestamp"
                    type = "TIMESTAMP"
                },
            ]
        )
        self_link           = "https://www.googleapis.com/bigquery/v2/projects/project-name/datasets/test_schema_update_dataset/tables/test_schema_update_table"
        table_id            = "test_schema_update_table"
        type                = "TABLE"
    }

3- Whenever you try updating again, this schema change is always shown

Terraform will perform the following actions:

  # google_bigquery_table.test_us_bigquery_table will be updated in-place
  ~ resource "google_bigquery_table" "test_us_bigquery_table" {
        creation_time       = 123
        dataset_id          = "test_schema_update_dataset"
        etag                = "123"
        expiration_time     = 0
        id                  = "project-name:test_schema_update_dataset.test_schema_update_table"
        labels              = {}
        last_modified_time  = 123
        location            = "US"
        num_bytes           = 0
        num_long_term_bytes = 0
        num_rows            = 0
        project             = "project-name"
      ~ schema              = jsonencode(
          ~ [
                {
                    mode = "REQUIRED"
                    name = "column_name"
                    type = "STRING"
                },
              ~ {
                  - mode = "REQUIRED" -> null
                  ~ name = "timestamp" -> "new_column"
                  ~ type = "TIMESTAMP" -> "FLOAT"
                },
              + {
                  + mode = "REQUIRED"
                  + name = "timestamp"
                  + type = "TIMESTAMP"
                },
                {
                    name = "new_column"
                    type = "FLOAT"
                },
            ]
        )
        self_link           = "https://www.googleapis.com/bigquery/v2/projects/project-name/datasets/test_schema_update_dataset/tables/test_schema_update_table"
        table_id            = "test_schema_update_table"
        type                = "TABLE"
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment