FesianXu

import os
import shutil
import argparse

def is_valid_filename(entry_bytes):
    try:
        decoded = entry_bytes.decode('utf-8')
    except UnicodeDecodeError:
        return False
    # 检查控制字符（ASCII 0-31，127）

FesianXu

INPUT_FILE=${1}
OUTPUT_PATH=${2}
NUM_PROC=${3}
tmp_files=".tmp"
rm -f ${tmp_files}/part-*

mkdir -p ${tmp_files}
mkdir -p ${OUTPUT_PATH}

total_lines=`wc -l ${INPUT_FILE} | cut -d " " -f1`

FesianXu

{
    "date": "20241226",
    "task_config": [
        "dummy_task_1",
        "dummy_task_2",
        "dummy_task_3"
    ],
    "wake_up_time": 600
}

FesianXu

🤗Huggingface Model Downloader

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, this command-line tool smartly utilizes wget or aria2 for LFS files and git clone for the rest.

Features

• ⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.
• 🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.
• 🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).
• 🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.
• 🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.