Skip to content

Instantly share code, notes, and snippets.

View knowsuchagency's full-sized avatar
💭
hustlin'

Stephan Fitzpatrick knowsuchagency

💭
hustlin'
View GitHub Profile
@inchoate
inchoate / including_external_package_in_dataflow.md
Last active February 2, 2024 11:40
Adding an extra package to a Python Dataflow project to run on GCP

The Problem

The documentation for how to deploy a pipeline with extra, non-PyPi, pure Python packages on GCP is missing some detail. This gist shows how to package and deploy an external pure-Python, non-PyPi dependency to a managed dataflow pipeline on GCP.

TL;DR: You external package needs to be a python (source/binary) distro properly packaged and shipped alongside your pipeline. It is not enough to only specify a tar file with a setup.py.

Preparing the External Package

Your external package must have a proper setup.py. What follow is an example setup.py for our ETL package. This is used to package version 1.1.1 of the etl library. The library requires 3 native PyPi packages to run. These are specified in the install_requires field. This package also ships with custom external JSON data, declared in the package_data section. Last, the setuptools.find_packages function searches for all available packages and returns that

@evansde77
evansde77 / mock_requests.py
Last active January 31, 2024 08:49
Example of mocking requests calls
#!/usr/bin/env python
"""
mocking requests calls
"""
import mock
import unittest
import requests
from requests.exceptions import HTTPError
@peterhurford
peterhurford / pytest-fixture-modularization.md
Created July 28, 2016 15:48
How to modularize your py.test fixtures

Using py.test is great and the support for test fixtures is pretty awesome. However, in order to share your fixtures across your entire module, py.test suggests you define all your fixtures within one single conftest.py file. This is impractical if you have a large quantity of fixtures -- for better organization and readibility, you would much rather define your fixtures across multiple, well-named files. But how do you do that? ...No one on the internet seemed to know.

Turns out, however, you can define fixtures in individual files like this:

tests/fixtures/add.py

import pytest

@pytest.fixture
@stinger
stinger / Swift3GCDURLSessionDataTask.swift
Last active March 26, 2018 21:36
Swift 3: GCD and URLSessionDataTask with a completionHandler
//: # Swift 3: CGD and URLSessionDataTask
import Foundation
import PlaygroundSupport
//: ### Create background queue
let queue = DispatchQueue.global(qos: .background)
//: ### Computed variable
var time:DispatchTime! {
@jbn
jbn / aiohttp_uptime_http_server.py
Created April 7, 2016 12:24
An example showing how to stream HTML in a aiohttp server.
import asyncio
from aiohttp import web
import subprocess
async def uptime_handler(request):
# http://HOST:PORT/?interval=90
interval = int(request.GET.get('interval', 1))
# Without the Content-Type, most (all?) browsers will not render
@lukas-h
lukas-h / license-badges.md
Last active March 28, 2024 10:06
Markdown License Badges for your Project

Markdown License badges

Collection of License badges for your Project's README file.
This list includes the most common open source and open data licenses.
Easily copy and paste the code under the badges into your Markdown files.

Notes

  • The badges do not fully replace the license informations for your projects, they are only emblems for the README, that the user can see the License at first glance.

Translations: (No guarantee that the translations are up-to-date)

@erichrobinson
erichrobinson / README.md
Last active February 24, 2024 08:32
SwitchResX Configuration

#SwitchResX Settings for LG 21:9 UltraWide

SwitchResX is a utility that allows users to override the default resolution settings in OSX. For more information, including download links, vist http://www.madrau.com/ .

##Disabling System Integrity Protection (SIP)

If you are running OSX 10.11 or higher, SIP must be disabled. To disable SIP do the following:

  • Boot into the recovery partition by pressing CMD + R when starting up your Mac.
  • Once in recovery mode, open a terminal window.
  • Type the command csrutil disable
@olivierlacan
olivierlacan / migrate_postgresql_database.md
Last active March 24, 2022 20:30
How to migrate a Homebrew-installed PostgreSQL database to a new major version (9.3 to 9.4) on OS X. See upgraded version of this guide: http://olivierlacan.com/posts/migrating-homebrew-postgres-to-a-new-version/

This guide assumes that you recently run brew upgrade postgresql and discovered to your dismay that you accidentally bumped from one major version to another: say 9.3.x to 9.4.x. Yes, that is a major version bump in PG land.

First let's check something.

brew info postgresql

The top of what gets printed as a result is the most important:

@gdamjan
gdamjan / client.py
Last active November 10, 2019 20:22
Python 3.5 async/await with aiohttp, parallel or sequential
import aiohttp
import asyncio
async def get_body(url):
response = await aiohttp.request('GET', url)
raw_html = await response.read()
return raw_html
async def main():
# run them sequentially (but the loop can do other stuff in the meanwhile)