Skip to content

Instantly share code, notes, and snippets.

View AndySze's full-sized avatar
🥇
Working hard

Andy Si AndySze

🥇
Working hard
  • Earth
View GitHub Profile
@GilesBathgate
GilesBathgate / preprocess.py
Last active May 17, 2024 09:09
Preprocess tinystories for Llama.c
import glob
import json
import os
import multiprocessing
from functools import partial
import random
import re
DATA_CACHE_DIR = "data"
BOYS_NAMES = ["Aaron","Abel","Abigail","Acorn","Adam","Aidan","Aiden","Al","Alan","Albert","Alby","Alex","Alexander","Alfie","Alfred","Ali","Allen","Amos","Andre","Andrew","Andy","Anthony","Antonio","Archie","Arthur","Avi","Bae","Bailey","Barry","Bart","Baxter","Ben","Benjamin","Benji","Bennie","Benny","Bert","Bill","Billy","Blake","Blaze","Bo","Bob","Bobbie","Bobby","Bobo","Boris","Boy","Brad","Braden","Brady","Brayden","Brent","Brett","Brian","Brix","Brody","Bruce","Bruno","Bryan","Bryce","Bubba","Bud","Budd","Bumpkin","Burt","Buster","Buzzy","Cai","Caleb","Calvin","Cape","Captain","Carl","Carlos","Carter","Cat","Chad","Charles","Charlie","Chen","Chet","Chip","Chris","Christopher","Chuck","Clifford","Clive","Clyde","Cody","Cole","Colin","Connor","Conor","Cooper","Corey","Craig","Dale","Damien","Dan","Daniel","Danny","Dare","Darren","Darryl","Dave","Davey","David","Davy","De
@padeoe
padeoe / README_hfd.md
Last active July 9, 2024 02:36
CLI-Tool for download Huggingface models and datasets with aria2/wget+git

🤗Huggingface Model Downloader

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, this command-line tool smartly utilizes wget or aria2 for LFS files and git clone for the rest.

Features

  • ⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.
  • 🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.
  • 🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).
  • 🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.
  • 🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.
@theycallmeloki
theycallmeloki / Raspberry Pi 4 x 6 cluster 65B Llama runner gist
Last active May 28, 2024 11:43
an ansbile script that quickly bootstraps a Pi 4 cluster with tools required to run a 65B llama model
You will need the following two files, be sure to change the IP address in both sections, the `hostfile_contents`
as well as the ansible inventory section to reflect your configuration
Rest of the steps should run parallely across the different machines, for any updates to this script,
check this thread: https://github.com/ggerganov/llama.cpp/issues/2164#issuecomment-1639640846
If you want an automated way to do this in the future and would like an opionated setup procedure,
check this repo: https://github.com/theycallmeloki/edith-cli
/*
Mesh Simplification
(C) by Sven Forstmann in 2014
derived from: https://github.com/sp4cerat/Fast-Quadric-Mesh-Simplification
and: https://github.com/timknip/mesh-decimate/blob/master/src/simplify.js
License : MIT
http://opensource.org/licenses/MIT
@jmtatsch
jmtatsch / gist:44e7c1a55fc29432254bdc61559d91c7
Last active April 12, 2018 05:10
ROS Kinetic on Sierra Work in Progress
# NOTE: These instructions do not represent a robust, self-troubleshooting install; they
# are definitely not suitable for dumping to a giant script and running as one. If you
# use them, they should be run one at a time, with an eye out for errors or problems
# along the way.
#
# The #1 issue you are likely to encounter is with Homebrew or Python packages whose
# binary components link against system Python. This will result in runtime segfaults,
# especially in rviz. If you suspect this is occurring, you can attempt to remove and
# reinstall the offending packages, or go for the nuclear option--- empty your Cellar
# and site-packages folders and start over with brewed python from the beginning.
@ajfisher
ajfisher / firmatatest.js
Last active December 22, 2019 08:15
Serial comms over hardware UART for Johnny-Five between Arduino and a Raspberry Pi
var firmata = require('firmata');
var repl = require('repl');
var board = new firmata.Board('/dev/ttyAMA0',function(err){
//arduino is ready to communicate
if (err) {
console.log("err:" + err);
return;
}
console.log("Firmata Firing LEDs");
@elvisimprsntr
elvisimprsntr / siriproxy-raspberrypi.txt
Last active January 4, 2019 00:45
SiriProxy running on a Raspberry Pi computer.
# $Header: /root/RCS/siriproxy-raspberrypi.txt,v 1.9 2013/11/09 08:11:00 root Exp $
# $Log: siriproxy-raspberrypi.txt,v $
# Revision 1.9 2013/11/09 08:11:00 root
# updated for latest RVM which installs dependencies and Ruby by default
#
# Revision 1.8 2013/03/22 20:41:20 root
# updated for Ruby 2.0.0 and SiriProxy 0.5.2
#
# Revision 1.7 2012/12/31 04:42:57 root
# shorter URL for newark and typos
@javiervidal
javiervidal / gist:1433880
Created December 5, 2011 15:05
To access url helpers (url_for, etc) from Rails console (Rails 3)
include Rails.application.routes.url_helpers
default_url_options[:host] = "localhost"
@watson
watson / ability.rb
Created October 5, 2011 09:50
Active Admin CanCan integration with shared front/backend User model and multi-level autherization
# app/models/ability.rb
# All front end users are authorized using this class
class Ability
include CanCan::Ability
def initialize(user)
user ||= User.new
can :read, :all
@Schnouki
Schnouki / movmd.py
Created January 12, 2011 00:57
A small metadata parser for MP4 containers, designed to edit dates saved by digital cameras on .mov videos.
#!/usr/bin/env python3
# -*- mode: python -*-
# This program is free software. It comes without any warranty, to the extent
# permitted by applicable law. You can redistribute it and/or modify it under
# the terms of the Do What The Fuck You Want To Public License, Version 2, as
# published by Sam Hocevar. See http://sam.zoy.org/wtfpl/COPYING for more
# details.
# Some useful resources: