padeoe

🤗Huggingface Model Downloader

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, this command-line tool smartly utilizes wget or aria2 for LFS files and git clone for the rest.

Features

• ⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.
• 🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.
• 🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).
• 🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.
• 🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.

geekodour

This is comparison between whisper.cpp and faster-whisper. The faster-whisper readme has some benchmarks on the readme but wanted to test it myself. For whisper, I just ran manually. For faster-whisper, wrote this small script

• https://github.com/guillaumekln/faster-whisper/
• https://github.com/ggerganov/whisper.cpp
• quanitization w faster-whisper: https://opennmt.net/CTranslate2/quantization.html
• quantization w whisper.cpp: https://github.com/ggerganov/whisper.cpp#quantization

whisper.cpp

• ./main -bs 5 -p 2 -f steve2.wav -m models/ggml-small.en.bin
  • Total 8 CPU threads on my 12 core machine
  • -bs 2 : actually performs better about 10s faster.

mattiasarro

# Taken from https://johanwind.github.io/2023/03/23/rwkv_details.html. 
# I've added additional comments restructured it a tiny bit, which makes it clearer for me.

import numpy as np
from torch import load as torch_load  # Only for loading the model weights
from tokenizers import Tokenizer

exp = np.exp
layer_norm = lambda x, w, b : (x - np.mean(x)) / np.std(x) * w + b
sigmoid = lambda x : 1/(1 + exp(-x))

ymoslem

# This example uses M2M-100 models converted to the CTranslate2 format.
# Download CTranslate2 models:
# • M2M-100 418M-parameter model: https://bit.ly/33fM1AO
# • M2M-100 1.2B-parameter model: https://bit.ly/3GYiaed


import ctranslate2
import sentencepiece as spm

tommyip

# Based on https://gist.github.com/bastibe/c0950e463ffdfdfada7adf149ae77c6f
# Changes:
# * Instead of overriding cd, we detect directory change. This allows the script to work
#   for other means of cd, such as z.
# * Update syntax to work with new versions of fish.
# * Handle virtualenvs that are not located in the root of a git directory.

function __auto_source_venv --on-variable PWD --description "Activate/Deactivate virtualenv on directory change"
  status --is-command-substitution; and return

SteveHere

import logging
import socks # use pysocks
import asyncio
from datetime import datetime
from itertools import cycle

logging.basicConfig(level=logging.INFO)
socks_router_loop = cycle((  # simple round-robin router to socks proxies
    ('127.2.0.0', 9050, None, None),  # address, port, username, password
    # ('127.3.0.0', 9050, "proxy", "passwordpassword"),

cristianadam

# MIT License
# 
# Copyright (c) 2019 Cristian Adam
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

zengxs

import asyncio
import logging
import re
from asyncio import StreamReader, StreamWriter, StreamReaderProtocol
from collections import namedtuple
from typing import Optional

import socks  # use pysocks

logging.basicConfig(level=logging.INFO)

dteoh

[merge]
	tool = vimdiff

[mergetool]
	keepBackup = false
    
[mergetool "vimdiff"]
	cmd = nvim -d $LOCAL $REMOTE $MERGED -c '$wincmd w' -c 'wincmd J'

W4ngatang

''' Script for downloading all GLUE data.

Note: for legal reasons, we are unable to host MRPC.
You can either use the version hosted by the SentEval team, which is already tokenized, 
or you can download the original data from (https://download.microsoft.com/download/D/4/6/D46FF87A-F6B9-4252-AA8B-3604ED519838/MSRParaphraseCorpus.msi) and extract the data from it manually.
For Windows users, you can run the .msi file. For Mac and Linux users, consider an external library such as 'cabextract' (see below for an example).
You should then rename and place specific files in a folder (see below for an example).

mkdir MRPC
cabextract MSRParaphraseCorpus.msi -d MRPC