slopp

 from gql import Client, gql
from gql.transport.requests import RequestsHTTPTransport
import os
import pandas as pd
from datetime import datetime, timedelta

def get_month_starts(start_date: datetime, end_date:datetime):
    """ Attributed to chatgpt """
    month_starts = []
    current_date = start_date.replace(day=1)

slopp

FROM python:3.10-slim

# Get the files that define dependencies
COPY *setup.py *requirements.txt /

# If setup.py exists, we install the dependencies before
# copying all other files
RUN if [ -f "setup.py" ]; then \
        pip install .; \
    fi

slopp

from dagster import asset, Definitions, AssetIn

assets_to_make = ["a", "b", "c"]

def make_assets(asset_to_make):
    @asset(
        name=asset_to_make
    )
    def asset_template():
        print(asset_to_make)

slopp

Push Metadata to External Postgres

Dagster maintains a metadata DB that allows users to filter, search, and take action on the status of past runs. However, in some cases it is useful to view this metadata outside of Dagster. This example shows how a scheduled Dagster job can be used to push the run metadata into another Postgres DB.

Get Started

To run this example:

1. Create a Postgres DB to be the metadata target

slopp

import json
from dagster import asset, AssetIn, AssetKey, Definitions, with_resources
from dagster_fivetran import build_fivetran_assets, fivetran_resource
from databricks_cli.sdk import JobService 
from dagster_databricks import databricks_client



fivetran_instance = fivetran_resource.configured(
    {

slopp

# Sigh - either asset config or dynamic partitions or asset factories depending on the use case 

slopp

Dagster: Controlling Parallelism within a Run

Dagster has many ways to control parallelism. In Dagster Cloud deployments, you can control how many concurrent runs can happen at one time through deployment settings.

Within a run, you can also control how many parallel operations happen at once. By default, runs use the multi-process executor, and the number of parallel operations within a run is based on the number of parallel threads available. For example, if you are using Dagster Cloud Hybrid with Kubernetes, the number of parallel operations within a run will be based on the resources available in a pod.

This behavior can be changed by modifiying the executor settings.

Default

slopp

from dagster import Definitions, job, resource, op, AssetMaterialization, asset


class MyResource():
    def hello(self, name: str): 
        print(f"hello {name}")

@resource
def my_resource(_):

slopp

What is a robust data platform?

This guide centralizes concepts needed to run a "robust" production data platform using Dagster Cloud, where robust means assets and infrastructure are:

• Fault Tolerant via replication, resource constraints, retries, parallelization, and run queues and priorities
• Observable via customizable logs and useful alerts

This guide does not cover every aspect of a production data platform. Other useful resources include:

• Testing and CICD to ensure new code does what is expected without breaking existing assets
• Project Structure to build a code base that can scale across teams and dependencies [Todo: Link to guide]
• Data Expectations to ensure the data flowing through your pipelines is valid and meets your expectations [Todo: refresh guide for assets, add section on conditional behavior]

slopp

Self-Dependent Asset Partitions

As of 1.1.7, Dagster supports assets that rely on prior versions of themselves, for example, an asset that implements a differential equation.

Getting Started

To run this example, first install the dependencies:

pip install dagster, dagit