Skip to content

Instantly share code, notes, and snippets.

@redsquare
Created November 3, 2022 10:24
Show Gist options
  • Save redsquare/69394ee2c13dc1a541f84d88a5b12665 to your computer and use it in GitHub Desktop.
Save redsquare/69394ee2c13dc1a541f84d88a5b12665 to your computer and use it in GitHub Desktop.
example prefect deploy - deploys new/changed flows
name: Register & Deploy Prefect flows
on:
push:
workflow_dispatch:
inputs:
pattern:
description: "Project Name or prohect name Pattern? (defaults to all)"
required: false
default: ".*"
permissions:
contents: read
jobs:
get-changed-flows-to-deploy:
runs-on: ubuntu-latest
outputs:
environment: ${{ steps.get-branch.outputs.environment }}
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: checkout repo
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: get branch and set environment for gh secrets
id: get-branch
run: |
echo "Running on branch ${{ github.ref }}"
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
echo "environment=production" >> $GITHUB_OUTPUT
elif [ "${{ github.ref }}" = "refs/heads/uat" ]; then
echo "environment=uat" >> $GITHUB_OUTPUT
else
echo "environment=test" >> $GITHUB_OUTPUT
fi
- name: get changed files
id: changed-files
uses: tj-actions/changed-files@v33
with:
json: 'true'
dir_names: 'true'
files: |
projects/**
#set matrix to be all the folders inside /projects that have changed
- id: set-matrix
name: set matrix of changed files
run: |
echo "matrix=${{ steps.changed-files.outputs.all_changed_files }}" >> $GITHUB_OUTPUT
deploy-flows:
needs: get-changed-flows-to-deploy
runs-on: ubuntu-latest
container:
image: prefecthq/prefect:2.6.5-python3.10
if: ${{ needs.get-changed-flows-to-deploy.outputs.matrix != '' && needs.get-changed-flows-to-deploy.outputs.matrix != '[]' }}
strategy:
matrix:
folder: ${{ fromJSON(needs.get-changed-flows-to-deploy.outputs.matrix) }}
max-parallel: 3
environment:
name: ${{ needs.get-changed-flows-to-deploy.outputs.environment }}
env:
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_S3_BLOCK: ${{ secrets.PREFECT_S3_BLOCK }}
PREFECT_WORK_QUEUE: ${{ secrets.PREFECT_WORK_QUEUE }}
PREFECT_K8_JOB_BLOCK: ${{ secrets.PREFECT_K8_JOB_BLOCK }}
steps:
- id: checkout-repo
name: checkout repo
uses: actions/checkout@v3
- id: get-current-folder
name: echo current folder
run: |
echo "${{ matrix.folder }}"
- id: get-envrionment
name: echo environment
shell: bash
run: echo ${{ needs.establish-environment-job.outputs.environment }}
- id: install-python-dependencies
name: install python dependencies
run: |
pip install s3fs
- id: prefect-cloud-login
name: prefect login
run: |
prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_KEY }}
prefect work-queue ls
- id: install-flow-requirments
name: install flow python dependencies
run: |
cd ${{ matrix.folder }}
pip install -r requirements.txt
- id: deploy-flows
name: prefect deploy flows
run: |
cd ${{ matrix.folder }}
find -name "*.deployment.py" | while read fname; do
python $fname
echo "deploying $fname"
done
echo "############### done ##############"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment