Skip to content

Instantly share code, notes, and snippets.

View shin1103's full-sized avatar

Shin-ichi Hashiba shin1103

  • Tokyo, Japan
View GitHub Profile
@shin1103
shin1103 / gist:29efa812dd46811ef30778592a0f5aee
Last active April 11, 2024 01:24
openmetadata_single_source_of_truth
https://openmetadata.slack.com/archives/C02B6955S4S/p1709558609380039
Hey, guys.
Question about general description metadata ingestion.
So, let's use as an example the PostgreSQL source. We may want to add/change descriptions of tables/columns on PostgreSQL,
which will reflect on OMD after enabling the metadata ingestion.
If we change the description later on OMD,
it seems that new metadata ingestions won't overwrite the changed description to the former/PostgreSQL value.
If we update descriptions on PostgreSQL that were changed on OMD, it will overwrite the changed on in OMD?
I've read that we should use OMD as the Source of Truth,
so it's not advisable to actually interact with the source to update the metadata which will further reflect on OMD via metadata ingestion?
import traceback
from typing import TYPE_CHECKING, Any, List, Optional
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
if TYPE_CHECKING:
from airflow.utils.context import Context
@shin1103
shin1103 / postgres_view_metadata_extractor.py
Created June 24, 2022 06:09
Postgres metadata extractor for Amundsen
from typing import ( # noqa: F401
Any, Dict, Iterator, Union,
)
from pyhocon import ConfigFactory, ConfigTree # noqa: F401
from databuilder.extractor.base_postgres_metadata_extractor import BasePostgresMetadataExtractor
class PostgresViewMetadataExtractor(BasePostgresMetadataExtractor):