This example shows the psuedo-code for a Dagster pipeline that:
- Accepts the path to a raw dataset as a string
- Runs a step to break the raw dataset into partitions
- For each partition, the pipeline runs a series of two processing steps. Each processing step is a call out to a Docker container to run supplying the partition key as an input argument. The partitions are run together in parallel before being collected in a final processing step that operates on all the partitions.
To run the pipeline: