Skip to content

Instantly share code, notes, and snippets.

View soldni's full-sized avatar
🏳️‍🌈
vibing!

Luca Soldaini soldni

🏳️‍🌈
vibing!
View GitHub Profile
import asyncio
import os
import random
from collections import deque
import ssl
from urllib.parse import urlparse
import aiohttp
import polars as pl
import aiofiles
@padeoe
padeoe / README_hfd.md
Last active June 18, 2024 18:51
CLI-Tool for download Huggingface models and datasets with aria2/wget+git

🤗Huggingface Model Downloader

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, this command-line tool smartly utilizes wget or aria2 for LFS files and git clone for the rest.

Features

  • ⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.
  • 🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.
  • 🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).
  • 🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.
  • 🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.
@ibeauregard
ibeauregard / exhaustive-ballot.ipynb
Created October 31, 2020 17:27
Exhaustive Ballot.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@lizthegrey
lizthegrey / attributes.rb
Last active February 24, 2024 14:11
Hardening SSH with 2fa
default['sshd']['sshd_config']['AuthenticationMethods'] = 'publickey,keyboard-interactive:pam'
default['sshd']['sshd_config']['ChallengeResponseAuthentication'] = 'yes'
default['sshd']['sshd_config']['PasswordAuthentication'] = 'no'
@dirkgr
dirkgr / logging_tqdm.py
Last active November 14, 2023 04:54
A drop-in replacement for TQDM which writes log messages instead of progress bars
from typing import *
import time
import logging
def logging_tqdm(
i,
*,
logger: Optional[logging.Logger] = None,
desc: str = "Working",
total: Optional[int] = None,
import json
from twitter import *
# Twitter app credentials
token = "..."
token_secret = "..."
consumer_key = "..."
consumer_secret = "..."
# Twitter client
@xiaom
xiaom / install_mosh_locally.sh
Last active April 2, 2024 16:48
install mosh locally
#!/bin/sh
# this script does absolutely ZERO error checking. however, it worked
# for me on a RHEL 6.3 machine on 2012-08-08. clearly, the version numbers
# and/or URLs should be made variables. cheers, zmil...@cs.wisc.edu
mkdir mosh
cd mosh