Skip to content

Instantly share code, notes, and snippets.

@jcrist

jcrist/dask_cloudprovider.py

Created December 16, 2020 18:05
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save jcrist/2d77ff27017f8591d1f478eca738bb65 to your computer and use it in GitHub Desktop.
Save jcrist/2d77ff27017f8591d1f478eca738bb65 to your computer and use it in GitHub Desktop.
Download ZIP
Prefect Example using DaskExecutor with dask-cloudprovider
Raw
dask_cloudprovider.py
from prefect import Flow
from prefect.executors import DaskExecutor
with Flow("daskcloudprovider-example") as flow:
# Add tasks to flow here...
# Execute this flow on a Dask cluster deployed on AWS Fargate
flow.executor = DaskExecutor(
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={"image": "prefecthq/prefect", "n_workers": 5}
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment