gabcoyne

service: flow-starter-lambda

frameworkVersion: "3"

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
  iamRoleStatements:
    - Effect: Allow

gabcoyne

flow-starter-lambda:$ sls deploy --stage dev --verbose
Running "serverless" from node_modules
Deploying flow-starter-lambda to stage dev (us-east-1)
Packaging
Generating requirements.txt from "pyproject.toml"
Parsed requirements.txt from pyproject.toml in /Users/gcoyne/src/personal/blogs/flow-starter-lambda/.serverless/requirements.txt
Installing requirements from "/Users/gcoyne/Library/Caches/serverless-python-requirements/3caa8622b77aad6da640d49705c6a65a8d4cd4b59ec3b6084e33747ba08c1dc7_x86_64_slspyc/requirements.txt"
Docker Image: lambci/lambda:build-python3.8
Using download cache directory /Users/gcoyne/Library/Caches/serverless-python-requirements/downloadCacheslspyc
Running docker run --rm -v /Users/gcoyne/Library/Caches/serverless-python-requirements/3caa8622b77aad6da640d49705c6a65a8d4cd4b59ec3b6084e33747ba08c1dc7_x86_64_slspyc\:/var/task\:z -v /Users/gcoyne/Library/Caches/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.8 python3.8 -m pip in

gabcoyne

[tool.poetry]
name = "flowrun-lambda"
version = "0.1.0"
description = ""
authors = ["George Coyne <@gabcoyne>"]

[tool.poetry.dependencies]
python = ">=3.8,<3.10"
prefect = "^0.15.3"

gabcoyne

import json
import logging
import os
import typing
from datetime import datetime

import boto3
import prefect
from prefect.run_configs.kubernetes import KubernetesRun

gabcoyne

import json
import logging
import os
import typing
from datetime import datetime

import boto3
import prefect
from prefect.run_configs.kubernetes import KubernetesRun

gabcoyne

from dask_kubernetes import KubeCluster, make_pod_spec
from prefect.storage import Docker
from prefect import task, Flow
from prefect.executors import DaskExecutor

# Configure a storage object, by default prefect's latest image will be used
storage = Docker(
    base_image="prefecthq/prefect:0.15.1-python3.7",
    python_dependencies=[
        "dask-kubernetes==2021.3.1",

gabcoyne

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  name: prefect-agent
spec:
  replicas: 1
  selector:

gabcoyne

Keybase proof

I hereby claim:

• I am gabcoyne on github.
• I am gabcoyne (https://keybase.io/gabcoyne) on keybase.
• I have a public key ASAIt3dHkKb61-jLQppfQePrQJFGFoU-Lkag20SCIeZxqwo

To claim this, I am signing this object:

gabcoyne

from prefect import task
from prefect.client import Secret
import pygit2


@task(name="Clone DBT")
def pull_dbt_repo():
    logger = prefect.context.get("logger")
    shutil.rmtree("dbt", ignore_errors=True)  # Delete folder on run
    git_token = Secret("GITHUB_ACCESS_TOKEN").get()