Skip to content

Instantly share code, notes, and snippets.

View garethbrickman's full-sized avatar

Gareth Brickman garethbrickman

View GitHub Profile
@garethbrickman
garethbrickman / InstigationStateQuery.gql
Last active April 5, 2024 20:09
Dagster - example query for InstigationStateQuery
query InstigationStateQuery {
instigationStateOrError(instigationSelector: {
repositoryName: "__repository__",
repositoryLocationName: "data-eng-pipeline",
name: "orders_sensor"
}) {
__typename
... on InstigationState {
id
status
@garethbrickman
garethbrickman / RunsBySensorAndStatus.gql
Last active July 25, 2024 21:07
Dagster - example query for finding a sensor's most recent failed run
query RunsBySensorAndStatus($filter: RunsFilter!) {
runsOrError(filter: $filter, limit: 1) {
__typename
... on Runs {
results {
runId
jobName
status
startTime
endTime
@garethbrickman
garethbrickman / GetDagsterCloudUsers.gql
Last active July 25, 2024 21:13
Dagster - GraphQL query to get list of Dagster+ organization users
query GetDagsterCloudUsers {
usersOrError {
... on DagsterCloudUsersWithScopedPermissionGrants {
users {
id
user {
email
name
}
}
@garethbrickman
garethbrickman / notes.txt
Last active July 25, 2024 21:15
Dagster - example GraphQL query for ScheduleDryRun
Note that:
- repositoryName = name of the Repository (if none, use "__repository__")
- repositoryLocationName = name of the Code Location
If the Code location name is something like "foo@bar" then this represents {repository}@{location}
i.e. the repository name is "foo" and location name is "bar".
@garethbrickman
garethbrickman / find_runs_with_cursor.py
Last active July 25, 2024 21:12
Dagster - A Dagster+ GraphQL API script for finding runs based on cursor timestamp
import os
from gql import Client, gql
from gql.transport.requests import RequestsHTTPTransport
url = "https://yourorg.dagster.cloud/prod/graphql"
token = os.getenv(
"DAGSTER_CLOUD_USER_TOKEN"
) # a User Token generated from the Cloud Settings page in Dagster Cloud. Note: User Token, not Agent Token
@garethbrickman
garethbrickman / databricks_sql_query_example.py
Created June 24, 2024 17:12
Databricks - Example querying using Databricks SQL Connector for Python.
from databricks import sql
import os
# Retrieve environment variables
server_hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
access_token = os.getenv("DATABRICKS_TOKEN")
# Establish a connection to the Databricks SQL warehouse
with sql.connect(server_hostname=server_hostname, http_path=http_path, access_token=access_token) as connection:
@garethbrickman
garethbrickman / example.py
Created June 26, 2024 16:50
Dagster - example showing how to pass the return data from an upstream asset to a downstream asset
from dagster import asset, Definitions, AssetExecutionContext
@asset
def upstream_asset():
return "data produced by upstream_asset"
@asset()
def downstream_asset(context: AssetExecutionContext, upstream_asset: str):
context.log.info(f"processed {upstream_asset}")
return
@garethbrickman
garethbrickman / RunQuery.gql
Created June 26, 2024 22:20
Dagster - GraphQL query to get metadata about a Run and its steps
query RunQuery($runId: ID!) {
runOrError(runId: $runId) {
__typename
... on Run {
id
status
startTime
endTime
stepStats {
stepKey
@garethbrickman
garethbrickman / find_run_step_data.py
Created June 28, 2024 20:11
Dagster - A Dagster+ GraphQL API script for finding a run's step data using its ID
import os
from gql import Client, gql
from gql.transport.requests import RequestsHTTPTransport
# Define the endpoint URL and token
org_name = "your_org_name"
base_url = f"https://{org_name}.dagster.cloud/"
deployment_name = "your_deployment_name" # branch deployments use an ID
url = base_url + deployment_name + "/graphql"
token = os.getenv("DAGSTER_CLOUD_USER_TOKEN") # a User Token generated from the Organization settings page in Dagster+. Note: User Token, not Agent Token
@garethbrickman
garethbrickman / AssetQuery.gql
Created July 5, 2024 16:57
Dagster - GraphQL query to get data about multiple assets
query AssetQuery($assetKeys: [AssetKeyInput!]!) {
assetNodes(assetKeys: $assetKeys) {
id
assetKey {
path
}
groupName
}
}