schrockn

Upgrading to 0.3.0

This guide is a step-by-step guide for upgrading from dagster 0.2.x to 0.3.0. This represents a substantial upgrade in capabilities but also some breaking API changes. We'll detail them, provide context and reasoning, and instructions about how to upgrade.

Required API Changes

1. No more top level config subpackage.

Error:

schrockn

This guide is a step-by-step guide for upgrading from dagster 0.2.x to 0.3.0. This represents a substantial upgrade in capabilities but also some breaking API changes. We'll detail them, provide context and reasoning, and instructions about how to upgrade.

Required API Changes

1. No more top level config subpackage.

Error:

from dagster import (

schrockn

New Concepts in 0.3.0

The upgrade guide describes the changes you are require to make to install 0.3.0. This guide describes the changes you should make in order to use the latest capabilities. The new concepts take some getting used to, but are quite powerful.

Resources

In 0.2.0 the notion of resources were relatively informal. This is no longer true: They are now an officially supported abstraction. They break apart context creation into composable, reusable chunks of software.

Defining a Resource

schrockn

We propose a convention for creating repos to house containerized kernels

We would have a scaffolding script that creates:

project_name/
   requirements.txt # pip freeze
   Dockerfile # perhaps through repo2docker + custom scripts
   notebooks/ # empty folder
 kernelspecs/

schrockn

.
├── .DS_Store
├── .coverage
├── .pylintrc -> ../.pylintrc
├── .pytest_cache
│   └── v
│       └── cache
│           └── nodeids
├── .style.yapf -> ../.style.yapf
├── .tox

schrockn

.
├── .DS_Store
├── .coverage
├── .pylintrc -> ../.pylintrc
├── .pytest_cache
│   └── v
│       └── cache
│           └── nodeids
├── .style.yapf -> ../.style.yapf
├── .tox

schrockn

{
  "python.unitTest.pyTestArgs": [
    "dagster/dagster_tests/core_tests",
    "dagit/dagit_tests/test_graphql.py"
  ],
  "python.analysis.disabled": [
    "unresolved-import"
  ],
  "python.unitTest.pyTestEnabled": true,
  "python.pythonPath": "/Users/schrockn/code/venvs/dagster-3.7.1/bin/python",

schrockn

class Foo:
  pass
  

schrockn

from dagster import solid, pipeline, RepositoryDefinition

@solid
def hello_world_solid(context):
    context.log.info('Hello, World!')

@pipeline
def hello_world_pipeline():
    hello_world_solid() # DSL for building dependency graph
    

schrockn

from dagster import solid, pipeline, RepositoryDefinition

# pipeline from above

def define_hello_world_repo():
    return RepositoryDefinition(
        name='hello_world_repo', 
        pipeline_defs=[hello_world_pipeline]
    )