Skip to content

Instantly share code, notes, and snippets.

View jeffmylife's full-sized avatar
🌊
back to learning

Jeffrey Lemoine jeffmylife

🌊
back to learning
View GitHub Profile
@jeffmylife
jeffmylife / equity_sampler.py
Created January 4, 2024 21:20
Sampling without replacement to acheive the most balanced representation across skewed distributions
import pandas as pd
def sample_dataframe(df, date_column, N):
"""
Samples a pandas DataFrame to achieve a balanced representation across different date groups.
This function ensures that smaller date groups (those with counts less than an average size)
are fully represented in the sample. Larger date groups are sampled uniformly to contribute
towards a total sample size of N. If the initial sampling process results in a total sample size
@jeffmylife
jeffmylife / Makefile
Created May 20, 2023 23:14
Makefile with yaml configuration for AWS Batch Job project. Defines variables in Makefile based on a target's dependencies.
.phony: all
all: push
# configuration setup
PROJECT_FILE_NAME=project.yml
CONFIG := python3 -c "import yaml; import json; from pathlib import Path; print(json.dumps(yaml.safe_load(Path(\"$(PROJECT_FILE_NAME)\").read_text())))"
OS := $(shell python3 -c "import platform; print(platform.system())")
# defaults
@jeffmylife
jeffmylife / async_download_images.py
Last active May 12, 2023 19:28
Async GET request for images using aiohttp
import re
import asyncio
from io import BytesIO
import time
from pathlib import Path
from PIL import Image
import aiohttp
import requests
@jeffmylife
jeffmylife / autodockvina_steps.sh
Last active June 28, 2021 17:51
Getting autodock vina to work in MacOS 11 Big Sur
## NOTE: I wouldn't run this script by itself.
# install vina
tar xzvf autodock_vina_1_1_2_mac_64bit.tar.gz
cd autodock_vina_1_1_2_mac_catalina_64bit
./installer.sh
mv bin/vina /usr/local/bin
# test
@jeffmylife
jeffmylife / aliasinate.py
Created March 16, 2021 23:51
Standardizes duplicate id's with different equivalent names, aka aliases, to an easier object.
import itertools
from operator import itemgetter
def aliasinate(lst, sep=';') -> dict:
'''
Parameters
----------
lst : list
List of strings containing aliases in any order or combination.
@jeffmylife
jeffmylife / s3_to_rds.py
Last active November 18, 2020 23:26
Transfer a large csv file on S3 to RDS Serverless through lambda function.
import csv
import json
import os
import boto3
import botocore.response
from pprint import pprint
MINIMUN_REMAINING_TIME_MS = int(os.getenv('MINIMUM_REMAINING_TIME_MS') or 10000)
@jeffmylife
jeffmylife / routes.py
Last active October 6, 2020 00:19
Python dict to Javascipt JSON object with Jinja templating
@app.route("/testing")
def test_json():
some_dict = {
's':"some_stringy",
'bool':True,
'lst':[1,2,3],
}
return render_template("some.html", python_dict=some_dict)
@jeffmylife
jeffmylife / DynamoSession.py
Created October 1, 2020 23:29 — forked from awesomebjt/DynamoSession.py
Store your Flask sessions in DynamoDB - Python 3.5
from uuid import uuid4
from datetime import datetime, timedelta
from flask.sessions import SessionInterface, SessionMixin
from werkzeug.datastructures import CallbackDict
import boto3
class DynamoSession(CallbackDict, SessionMixin):
def __init__(self, initial=None, sid=None):
CallbackDict.__init__(self, initial)
@jeffmylife
jeffmylife / aws_fargate_docker_application_load_balancer_without_public_ip.md
Last active September 25, 2020 19:22 — forked from jonashaag/aws_fargate_docker_application_load_balancer_without_public_ip.md
AWS Fargate Docker Application Load Balancer How to (without public IP)

AWS Fargate Docker Simple Deployment Setup with SSL termination

How to:

  • create a Docker-based AWS Fargate/ECS deployment
  • without the Docker containers having a public IP
  • with an Application Load Balancer as reverse proxy / SSL termination proxy sitting in front of the containers

For Fargate/ECS to be able to access your Docker images hosted on ECR (or somewhere else) you'll have to allow outbound internet access to the Fargate subnets. Here's how you do it.

@jeffmylife
jeffmylife / s3_multipart_upload.py
Last active October 21, 2022 22:19 — forked from teasherm/s3_multipart_upload.py
S3 Multipart Upload from request.file
'''
See comments for description & usage instructions.
'''
import boto3, botocore
from .config import S3_KEY, S3_SECRET, S3_BUCKET, S3_LOCATION
def get_file_size(file):
file.seek(0, os.SEEK_END)