Skip to content

Instantly share code, notes, and snippets.

View jlertle's full-sized avatar

Jason Lee Ertle jlertle

  • Saint Petersburg, FL, US
View GitHub Profile
@jlertle
jlertle / README.md
Created January 29, 2019 02:32 — forked from wildmichael/README.md
superscript test

This is some superscript text.

@jlertle
jlertle / black.md
Created January 26, 2019 00:03 — forked from kstrauser/black.md
Using the "black" Python formatter in VS Code

This is how to use the Black Python code formatter in VS Code.

Make a Python 3.6 virtualenv for running Black

Black itself requires Python 3.6 to run, but few of our projects are on that version. The VS Code plugin conveniently lets you run black from its own virtualenv.

I had to give a specific version of black this morning. I didn't yesterday. Don't specify the version unless it makes you (and if you do, give the current version, not the one from this doc).

$ cd ~/Envs
$ git remote rm origin
$ git remote add origin git@github.com:aplikacjainfo/proj1.git
$ git config master.remote origin
$ git config master.merge refs/heads/master
@jlertle
jlertle / PySpark_Vectorized_UDFs.ipynb
Created October 8, 2018 08:05 — forked from BryanCutler/PySpark_Vectorized_UDFs.ipynb
PySpark vectorized UDFs with Arrow
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jlertle
jlertle / faster_toPandas.py
Created October 2, 2018 17:13 — forked from joshlk/faster_toPandas.py
PySpark faster toPandas using mapPartitions
import pandas as pd
def _map_to_pandas(rdds):
""" Needs to be here due to pickling issues """
return [pd.DataFrame(list(rdds))]
def toPandas(df, n_partitions=None):
"""
Returns the contents of `df` as a local `pandas.DataFrame` in a speedy fashion. The DataFrame is
repartitioned if `n_partitions` is passed.
@jlertle
jlertle / sagemaker_multiin_repro.py
Created September 18, 2018 06:06 — forked from zmjjmz/sagemaker_multiin_repro.py
Sagemaker Multi-input repro
import os
import json
import numpy
import tensorflow
from tensorflow.python.estimator.export.export import build_raw_serving_input_receiver_fn
print("Tensorflow version: {0}".format(tensorflow.VERSION))
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
@jlertle
jlertle / pyspark_udf_filtering.py
Created September 18, 2018 01:51 — forked from samuelsmal/pyspark_udf_filtering.py
PySpark DataFrame filtering using a UDF and Regex
from pyspark.sql.functions import udf
from pyspark.sql.types import BooleanType
def regex_filter(x):
regexs = ['.*ALLYOURBASEBELONGTOUS.*']
if x and x.strip():
for r in regexs:
if re.match(r, x, re.IGNORECASE):
return True
@jlertle
jlertle / tensorflow_mnist_estimator.py
Created September 13, 2018 04:20 — forked from alsrgv/tensorflow_mnist_estimator.py
Horovod with Estimator API
# Copyright 2017 Uber Technologies, Inc. All Rights Reserved.
# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
@jlertle
jlertle / crop_and_resize.py
Created February 16, 2018 23:39 — forked from mattjmorrison/crop_and_resize.py
Resize and Crop images with Python and PIL
from PIL import Image, ImageChops
def trim(im, border):
bg = Image.new(im.mode, im.size, border)
diff = ImageChops.difference(im, bg)
bbox = diff.getbbox()
if bbox:
return im.crop(bbox)
def create_thumbnail(path, size):
@jlertle
jlertle / gist:b75c9c2f17174806d0115a103e133609
Created January 14, 2018 06:56 — forked from anonymous/gist:10675250
Motion Blur + Chromatic Aberration
// by dave @ beesandbombs.tumblr.com >:)
void setup() {
setup_();
result = new int[width*height][3];
result_ = new int[width*height][3];
}
int[][] result, result_;
float time;