Skip to content

Instantly share code, notes, and snippets.

@sidravic

sidravic/dask_cloudprovider.py

Forked from jcrist/dask_cloudprovider.py
Created June 17, 2022 03:45
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sidravic/5282db2e12b83622bae7269744170f40 to your computer and use it in GitHub Desktop.
Save sidravic/5282db2e12b83622bae7269744170f40 to your computer and use it in GitHub Desktop.
Download ZIP
Prefect Example using DaskExecutor with dask-cloudprovider
Raw
dask_cloudprovider.py
from prefect import Flow
from prefect.executors import DaskExecutor
with Flow("daskcloudprovider-example") as flow:
# Add tasks to flow here...
# Execute this flow on a Dask cluster deployed on AWS Fargate
flow.executor = DaskExecutor(
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={"image": "prefecthq/prefect", "n_workers": 5}
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment