palewire

How to deploy a Prefect agent to Google Kubernetes Engine

This post contains code and commands you can use to deploy Prefect agents to Google Cloud’s Google Kubernetes Engine. The agents stand ready to execute workflows triggered by Prefect projects. One agent can run tasks from multiple projects.

The example here demonstrates how to create a single agent with minimal customization. It is configured with a Dockerfile, which installs necessary dependencies, and a k8s.cfg file, which connects the system to a Prefect account.

Agents are deployed via the gcloud command-line utility and its kubectl extension. Proper permissions within a project on Google Cloud are required.

Getting started

TylerWanner

Provisioning Prefect Server on GCP using Terraform

Requirements: Terraform, GCP credentials with permission to create a project Tested with: Terraform v0.12.23, google cloud provider v3.26

To run:

You can use an environment variable to set which service account key to use during provisioning export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key The service account you use must belong to a GCP project that has the necessary APIs enabled (such as billing and resource manager)--if it does not, you may have to enable these APIs manually along the way in those projects In this directory, run terraform init then terraform apply -- you will need to provide a billing_id and organization_id for your GCP project

Once successful, be sure to add the following to your ~/.prefect/config.toml

Vigrond

# Rate limiting with Celery + Django + Redis
# Multiple Fixed Windows Algorithm inspired by Figma https://www.figma.com/blog/an-alternative-approach-to-rate-limiting/
#   and Celery's sometimes ambiguous, vague, and one-paragraph documentation
#
# Celery's Task is subclassed and the is_rate_okay function is added


# celery.py or however your App is implemented in Django
import os
import math

jonlabelle

title	subtitle	author	date	source
Docker Compose Cheatsheet	Quick reference for Docker Compose commands and config files	Jon LaBelle	April 7, 2019	https://jonlabelle.com/snippets/view/markdown/docker-compose-cheatsheet

Basic config example

levidyrek

"""
Django ORM Optimization Tips

Caveats:
* Only use optimizations that obfuscate the code if you need to.
* Not all of these tips are hard and fast rules.
* Use your judgement to determine what improvements are appropriate for your code.
"""

# ---------------------------------------------------------------------------

LeCoupa

/* *******************************************************************************************
 * GLOBAL CONFIG
 * Vue.config is an object containing Vue’s global configurations.
 * You can modify its properties listed below before bootstrapping your application.
 * https://vuejs.org/v2/api/#Global-Config
 * ******************************************************************************************* */


// Configure whether to allow vue-devtools inspection
Vue.config.devtools = true

btoo

import threading
from queue import Queue
import time

print_lock = threading.Lock() # special lock we're using to make sure each worker won't pring until it's done with it job

def exampleJob(worker):
    time.sleep(0.5) # let's say a job takes .5s to finish

    with print_lock: # with the lock, print some info. when that's done, release the lock so another worker can print with it

aisayko

def bulk_upsert(model, fields, values, by):
    """
    Return the tuple of (inserted, updated) ids
    """
    result = (None, None)

    if values:
        stmt = """
        WITH data_set AS (
            INSERT INTO %s (%s)

vik-y

docker-compose cheat sheet

Reference: https://docs.docker.com/compose/compose-file/#compose-file-structure-and-examples

Collection of a few docker-compose examples which I use regularly

Instructions on using docker-compose files

# docker-compsoe version to be used must be mentioned at the top 
version: '3'
services:

Sniedes722

import os

import boto3
from flask import Flask, render_template, request, url_for, redirect

app = Flask(__name__)

AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
S3_BUCKET = os.environ['S3_BUCKET']