Skip to content

Instantly share code, notes, and snippets.

@elyast
elyast / sft_trainer.py
Created October 19, 2023 19:09 — forked from lewtun/sft_trainer.py
Fine-tuning Mistral 7B with TRL & DeepSpeed ZeRO-3
# This is a modified version of TRL's `SFTTrainer` example (https://github.com/huggingface/trl/blob/main/examples/scripts/sft_trainer.py),
# adapted to run with DeepSpeed ZeRO-3 and Mistral-7B-V1.0. The settings below were run on 1 node of 8 x A100 (80GB) GPUs.
#
# Usage:
# - Install the latest transformers & accelerate versions: `pip install -U transformers accelerate`
# - Install deepspeed: `pip install deepspeed==0.9.5`
# - Install TRL from main: pip install git+https://github.com/huggingface/trl.git
# - Clone the repo: git clone github.com/huggingface/trl.git
# - Copy this Gist into trl/examples/scripts
# - Run from root of trl repo with: accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero3.yaml --gradient_accumulation_steps 8 examples/scripts/sft_trainer.py
@elyast
elyast / front.html
Created December 8, 2016 23:11 — forked from randerzander/front.html
zeppelin angular-tricks
%angular
<input id="textbox" class="hide" ng-model="someAngularVar"></input>
<button id="btn" type="submit" onclick="update()">UpperCase It!</button>
<script type="text/javascript">
function update(){
var element = $('#textbox');
var currentVal = element.val();
//Update the value
element.val(currentVal.toUpperCase());
@elyast
elyast / gist.scala
Last active May 10, 2017 08:52
Shapeless usage in spark pipelines
// This allow to transform single entity (from a given RDD or DStream) in different ways
// using HList of mappers and bind them into given output type
// potentially useful when doing feature extraction out of single entity / basic etl
// Usage:
import twitter4j.Status
import shapeless._
val user: Status => String = ... // a function that extract author of tweet
@elyast
elyast / gist:1833492
Created February 15, 2012 05:26
brew install duplicity problem
➜ ~ brew install -v duplicity
==> Downloading http://launchpad.net/duplicity/0.6-series/0.6.17/+download/duplicity-0.6.17.tar.gz
/usr/bin/curl -qf#LA Homebrew 0.8.1 (Ruby 1.8.7-249; Mac OS X 10.6.8) http://launchpad.net/duplicity/0.6-series/0.6.17/+download/duplicity-0.6.17.tar.gz -o /Users/lukaszjastrzebski/Library/Caches/Homebrew/duplicity-0.6.17.tar.gz
######################################################################## 100,0%
/usr/bin/tar xf /Users/lukaszjastrzebski/Library/Caches/Homebrew/duplicity-0.6.17.tar.gz
==> python setup.py install --prefix=/usr/local/Cellar/duplicity/0.6.17 --install-purelib=/usr/local/Cellar/duplicity/0.6.17/libexec --install-platlib=/usr/local/Cellar/duplicity/0.6.17/libexec --install-scripts=/usr/local/Cellar/duplicity/0.6.17/bin
python setup.py install --prefix=/usr/local/Cellar/duplicity/0.6.17 --install-purelib=/usr/local/Cellar/duplicity/0.6.17/libexec --install-platlib=/usr/local/Cellar/duplicity/0.6.17/libexec --install-scripts=/usr/local/Cellar/duplicity/0.6.17/b