Skip to content

Instantly share code, notes, and snippets.

@mlabonne
Last active March 10, 2024 04:53
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save mlabonne/8055f6335e2b85f082c8c75561321a66 to your computer and use it in GitHub Desktop.
Save mlabonne/8055f6335e2b85f082c8c75561321a66 to your computer and use it in GitHub Desktop.
Axolotl config file to train mlabonne/EvolCodeLlama-7b (https://huggingface.co/mlabonne/EvolCodeLlama-7b)
base_model: codellama/CodeLlama-7b-hf
base_model_config: codellama/CodeLlama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
hub_model_id: EvolCodeLlama-7b
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: mlabonne/Evol-Instruct-Python-1k
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./qlora-out
adapter: qlora
lora_model_dir:
sequence_len: 2048
sample_packing: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: axolotl
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 10
num_epochs: 3
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
eval_steps: 0.01
save_strategy: epoch
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
@Rahmat711
Copy link

Rahmat711 commented Sep 7, 2023

Hi this is not running getting error /code_llma/axolotl/src/axolotl/utils/data.py", line 503, in load_prepare_datasets
train_fingerprint = hashlib.md5(
TypeError: openssl_md5() takes at most 1 argument (2 given)
Traceback (most recent call last):
Can you please help solving the error!
thanks in advance

@alan-m-thomas
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment