Skip to content

Instantly share code, notes, and snippets.

View lukeorland's full-sized avatar

Luke Orland lukeorland

View GitHub Profile
@lukeorland
lukeorland / fetch_flow_run_logs.py
Created March 30, 2023 19:39
Prefect Cloud 1.0 - fetch logs for flow run
"""
Retrieve all the logs for a flow run.
E.g.
FLOW_RUN_ID=adff99c1-7dcf-40c1-bd1d-1c026ad930cb \
python \
notes-synced/snippets/fetch_flow_run_logs.py \
print_concatenated_log_messages \
$FLOW_RUN_ID \
#! /usr/bin/env zsh
# https://stackoverflow.com/questions/7726949/remove-tracking-branches-no-longer-on-remote/38404202#comment91928557_38404202
function git-prune-not-fully-merged-branches {
git fetch
git checkout main
git branch -vv | grep ': gone]'| grep -v "\*" | awk '{ print $1; }' | xargs -r git branch -d
git checkout -
}
@lukeorland
lukeorland / keybase.md
Created September 21, 2020 00:30
Keybase proof

Keybase proof

I hereby claim:

  • I am lukeorland on github.
  • I am orluke (https://keybase.io/orluke) on keybase.
  • I have a public key ASAwfXXw2gairO6ye-QxD5TIuBCWX3Eq5Jr2y45GZMj7Iwo

To claim this, I am signing this object:

from prefect import Flow, Parameter
from prefect.tasks.secrets import PrefectSecret
aws_secret = PrefectSecret(name=None) # initializes Task, unrelated to binding it within a Flow
with Flow("parameter_to_secret") as flow:
aws_credentials_secret_name = Parameter(
'aws_credentials_secret_name', default='AWS_CREDENTIALS',
)
import prefect
from prefect import Flow, Parameter, task
@task
def func_factory():
def func():
prefect.context.get('logger').info('this is a func')
return func
@lukeorland
lukeorland / dir_containing_this.py
Created November 26, 2019 19:49
directory containing this Python file
import inspect
import os
def get_dir_containing_this_file():
return os.path.dirname(
os.path.abspath(
inspect.getframeinfo(
inspect.currentframe()).filename))
@lukeorland
lukeorland / multiprocessing_pandas_apply.py
Last active September 12, 2019 20:00
easy multiprocessing apply pandas
import multiprocessing
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
df['normalized_text'] = pool.map(normalize_text, df['text'])
pool.close()
pool.join()
@lukeorland
lukeorland / concurrent_ThreadPoolExecutor.py
Last active October 20, 2017 14:25
python3 concurrent ThreadPoolExecutor
#!/usr/bin/env python3
import concurrent.futures
import time
def slumber(seconds):
time.sleep(seconds)
print("slept %s seconds" % seconds)
# -*- coding: utf-8 -*-
import argparse
import csv
import re
fields = ("Start Time", "End Time", "Activity", "Text", "Notes")
def parse_arguments():
parser = argparse.ArgumentParser(description='')

Keybase proof

I hereby claim:

  • I am lukeorland on github.
  • I am orluke (https://keybase.io/orluke) on keybase.
  • I have a public key whose fingerprint is 50F6 A605 06CC 59DF 715B 1F0F 1307 0B06 3F6B E0F7

To claim this, I am signing this object: