Skip to content

Instantly share code, notes, and snippets.

View hayden-donnelly's full-sized avatar

Hayden Donnelly hayden-donnelly

View GitHub Profile
@hayden-donnelly
hayden-donnelly / images_to_hf_parquet.py
Created March 20, 2024 21:30
Script to convert a directory of images to a collection of Apache Parquet files with HuggingFace metadata.
# Example usage:
# python images_to_hf_parquet.py --input ./base_image_directory/ --output ./parquet_output_directory/ --samples_per_file 10000
import pyarrow as pa
import pyarrow.parquet as pq
from PIL import Image
import os, io, json, glob, argparse
def save_table(image_data, table_number, output_path, zfill_amount):
print(f'Entries in table {table_number}: {len(image_data)}')
@hayden-donnelly
hayden-donnelly / linear_attention.py
Created February 27, 2024 20:16
A JAX/Flax implementation of linear attention from "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention".
import jax
from jax import numpy as jnp
from flax import linen as nn
from typing import Any, Callable
def linear_attention(
query: jax.Array,
key: jax.Array,
value: jax.Array,
dtype: Any,
@hayden-donnelly
hayden-donnelly / make_crop_labels.py
Last active January 27, 2024 04:29
Script to make square crop labels for images.
# Example usage:
# python make_crop_labels.py --input_path data/images --output_path --data/cropped_images --csv_path data/crops.csv
# Controls:
# scroll to change crop size, mouse to aim the crop, left click to crop image and move to next, x to skip to the next image.
# The script is pretty messy since I quickly hacked it together with little regard for quality, but it works.
import pygame
import argparse
@hayden-donnelly
hayden-donnelly / data_filter.py
Last active December 10, 2023 10:41
Script to filter images from input directory to output directory
# Run with: python data_filter.py --input_path <input path> --output_path <output path>
# Controls: z to copy image into output directory, x to skip to next image.
import pygame
import argparse
import shutil
import os
import argparse
def load_image(path):