This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from datasets import load_dataset | |
from trl import SFTTrainer | |
from peft import LoraConfig | |
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments | |
tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-130m-hf") | |
model = AutoModelForCausalLM.from_pretrained("state-spaces/mamba-130m-hf") | |
dataset = load_dataset("Abirate/english_quotes", split="train") | |
training_args = TrainingArguments( | |
output_dir="./results", | |
num_train_epochs=3, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import sys | |
from subprocess import call | |
import json | |
import os | |
import requests | |
def download_gists(gists: list): | |
for gist in gists: | |
call(["git", "clone", gist["git_pull_url"]]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
""" | |
How to use multiple queues for passing data to/from a multiprocessing pool. | |
""" | |
from multiprocessing import Pool, Queue | |
import shortuuid | |
def f(q_in, q_out): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
""" | |
nsb_entropy.py | |
June, 2011 written by Sungho Hong, Computational Neuroscience Unit, Okinawa Institute of | |
Science and Technology | |
May, 2019 updated to Python 3 by Charlie Strauss, Los Alamos National Lab | |
This script is a python version of Mathematica functions by Christian Mendl |