I hereby claim:
- I am gabcoyne on github.
- I am gabcoyne (https://keybase.io/gabcoyne) on keybase.
- I have a public key ASAIt3dHkKb61-jLQppfQePrQJFGFoU-Lkag20SCIeZxqwo
To claim this, I am signing this object:
from prefect import task | |
from prefect.client import Secret | |
import pygit2 | |
@task(name="Clone DBT") | |
def pull_dbt_repo(): | |
logger = prefect.context.get("logger") | |
shutil.rmtree("dbt", ignore_errors=True) # Delete folder on run | |
git_token = Secret("GITHUB_ACCESS_TOKEN").get() |
I hereby claim:
To claim this, I am signing this object:
--- | |
apiVersion: apps/v1 | |
kind: Deployment | |
metadata: | |
labels: | |
app: prefect-agent | |
name: prefect-agent | |
spec: | |
replicas: 1 | |
selector: |
from dask_kubernetes import KubeCluster, make_pod_spec | |
from prefect.storage import Docker | |
from prefect import task, Flow | |
from prefect.executors import DaskExecutor | |
# Configure a storage object, by default prefect's latest image will be used | |
storage = Docker( | |
base_image="prefecthq/prefect:0.15.1-python3.7", | |
python_dependencies=[ | |
"dask-kubernetes==2021.3.1", |
import json | |
import logging | |
import os | |
import typing | |
from datetime import datetime | |
import boto3 | |
import prefect | |
from prefect.run_configs.kubernetes import KubernetesRun |
import json | |
import logging | |
import os | |
import typing | |
from datetime import datetime | |
import boto3 | |
import prefect | |
from prefect.run_configs.kubernetes import KubernetesRun |
[tool.poetry] | |
name = "flowrun-lambda" | |
version = "0.1.0" | |
description = "" | |
authors = ["George Coyne <@gabcoyne>"] | |
[tool.poetry.dependencies] | |
python = ">=3.8,<3.10" | |
prefect = "^0.15.3" |
flow-starter-lambda:$ sls deploy --stage dev --verbose | |
Running "serverless" from node_modules | |
Deploying flow-starter-lambda to stage dev (us-east-1) | |
Packaging | |
Generating requirements.txt from "pyproject.toml" | |
Parsed requirements.txt from pyproject.toml in /Users/gcoyne/src/personal/blogs/flow-starter-lambda/.serverless/requirements.txt | |
Installing requirements from "/Users/gcoyne/Library/Caches/serverless-python-requirements/3caa8622b77aad6da640d49705c6a65a8d4cd4b59ec3b6084e33747ba08c1dc7_x86_64_slspyc/requirements.txt" | |
Docker Image: lambci/lambda:build-python3.8 | |
Using download cache directory /Users/gcoyne/Library/Caches/serverless-python-requirements/downloadCacheslspyc | |
Running docker run --rm -v /Users/gcoyne/Library/Caches/serverless-python-requirements/3caa8622b77aad6da640d49705c6a65a8d4cd4b59ec3b6084e33747ba08c1dc7_x86_64_slspyc\:/var/task\:z -v /Users/gcoyne/Library/Caches/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.8 python3.8 -m pip in |
service: flow-starter-lambda | |
frameworkVersion: "3" | |
provider: | |
name: aws | |
runtime: python3.8 | |
lambdaHashingVersion: 20201221 | |
iamRoleStatements: | |
- Effect: Allow |