mmlu_branch execution log (ilab model evaluate --model models/tuned-0701-1954/samples_4992 --base-model models/ibm/granite-7b-base --benchmark mmlu_branch --sdg-path generated/lm_eval/tasks)

mmlubranch.log

INFO 2024-07-06 18:00:23,647 utils.py:145: _init_num_threads Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-06 18:00:23,647 utils.py:148: _init_num_threads Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-06 18:00:23,647 utils.py:161: _init_num_threads NumExpr defaulting to 16 threads.
INFO 2024-07-06 18:00:23,802 config.py:58: <module> PyTorch version 2.3.1 available.
INFO 2024-07-06 18:00:35,580 evaluator.py:152: simple_evaluate Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
INFO 2024-07-06 18:00:35,580 evaluator.py:189: simple_evaluate Initializing hf model, with arguments: {'pretrained': 'models/tuned-0701-1954/samples_4992', 'dtype': 'bfloat16'}
/usr/local/lib64/python3.11/site-packages/torch/cuda/__init__.py:619: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
INFO 2024-07-06 18:00:35,598 huggingface.py:170: __init__ Using device 'cuda'
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Generating test split: 27 examples [00:00, 1925.24 examples/s]
WARNING 2024-07-06 18:01:28,520 task.py:325: fewshot_docs [Task: tonsils] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
WARNING 2024-07-06 18:01:28,520 task.py:325: fewshot_docs [Task: tonsils] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
WARNING 2024-07-06 18:01:28,527 evaluator.py:251: simple_evaluate Overwriting default num_fewshot of tonsils from None to 2
INFO 2024-07-06 18:01:28,527 evaluator.py:261: simple_evaluate Setting fewshot random generator seed to 1234
INFO 2024-07-06 18:01:28,528 task.py:411: build_all_requests Building contexts for tonsils on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 27/27 [00:00<00:00, 301.63it/s]
INFO 2024-07-06 18:01:28,621 evaluator.py:438: evaluate Running loglikelihood requests
Running loglikelihood requests:   0%|                                                                                                                                                                           | 0/108 [00:00<?, ?it/s]We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
Running loglikelihood requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 108/108 [00:01<00:00, 57.01it/s]
WARNING 2024-07-06 18:01:30,615 huggingface.py:1315: get_model_sha Failed to get model SHA for models/tuned-0701-1954/samples_4992 at revision main. Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'models/tuned-0701-1954/samples_4992'. Use `repo_type` argument if needed.
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
INFO 2024-07-06 18:01:36,819 evaluator.py:152: simple_evaluate Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
INFO 2024-07-06 18:01:36,819 evaluator.py:189: simple_evaluate Initializing hf model, with arguments: {'pretrained': 'models/ibm/granite-7b-base', 'dtype': 'bfloat16'}
INFO 2024-07-06 18:01:36,821 huggingface.py:170: __init__ Using device 'cuda'
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [01:03<00:00, 10.55s/it]
WARNING 2024-07-06 18:02:40,358 task.py:325: fewshot_docs [Task: tonsils] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
WARNING 2024-07-06 18:02:40,358 task.py:325: fewshot_docs [Task: tonsils] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
WARNING 2024-07-06 18:02:40,363 evaluator.py:251: simple_evaluate Overwriting default num_fewshot of tonsils from None to 2
INFO 2024-07-06 18:02:40,363 evaluator.py:261: simple_evaluate Setting fewshot random generator seed to 1234
INFO 2024-07-06 18:02:40,364 task.py:411: build_all_requests Building contexts for tonsils on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 27/27 [00:00<00:00, 301.27it/s]
INFO 2024-07-06 18:02:40,457 evaluator.py:438: evaluate Running loglikelihood requests
Running loglikelihood requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 108/108 [00:00<00:00, 179.23it/s]
WARNING 2024-07-06 18:02:41,154 huggingface.py:1315: get_model_sha Failed to get model SHA for models/ibm/granite-7b-base at revision main. Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'models/ibm/granite-7b-base'. Use `repo_type` argument if needed.
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
# KNOWLEDGE EVALUATION REPORT

## BASE MODEL
models/ibm/granite-7b-base

## MODEL
models/tuned-0701-1954/samples_4992

### AVERAGE:
+0.04 (across 1)

### IMPROVEMENTS:
1. tonsils (+0.04)