Python script to create a fake manifest to defer to locally

README.MD

This script could be used locally when someone wants to build models in their own schema while deferring to the Prodcution models and not having the Production manifest file stored locally.

It creates a modified manifest.yml file based on a local one, changing the schema name from the local one to the Production one. The target_name argument is the name of the folder where the new manifest will be created. It defaults to target2 but can be updated

Usage: python build_defer_manifest.py [--target_name TARGET_NAME] dev_schema defer_schema

After having generated the manifest, you can do a dbt run -s <my_model> --defer --state target2

Notes: It currenlty assumes that the same DB is used for Dev and Prod but the script could be updated to allow switching DBs as well.

build_defer_manifest.py

import json
import os
import argparse

DEFAULT_TARGET_NAME = "target2"


parser = argparse.ArgumentParser(description="Generate a fake manifest for deferring")
parser.add_argument("dev_schema", help="the dev schema used in profiles.yml")
parser.add_argument("defer_schema", help="the schema used in Prod we want to defer to")
parser.add_argument(
    "--target_name", "-t", default=DEFAULT_TARGET_NAME, help="name of the target folder"
)


def main(target_name, dev_schema, defer_schema):
    # read from file
    with open("target/manifest.json", "r") as f:
        manifest = json.load(f)

    for node, node_vals in manifest["nodes"].items():
        if node_vals["resource_type"] in ["model", "snapshot", "seed"]:
            node_vals["schema"] = node_vals["schema"].replace(dev_schema, defer_schema)

    # create the new folder if it doesn't exist
    target_filename = f"{target_name}/manifest.json"
    os.makedirs(os.path.dirname(target_filename), exist_ok=True)

    # write file
    with open(target_filename, "w") as f:
        json.dump(manifest, f, indent=4)


# if main
if __name__ == "__main__":
    args = parser.parse_args()
    main(
        target_name=args.target_name,
        dev_schema=args.dev_schema,
        defer_schema=args.defer_schema,
    )