Skip to content

Instantly share code, notes, and snippets.

@eruo005-dev

eruo005-dev/llama-3.2-3b-instruct-Q4_K_M-turkishmmlu.tmp.log

Created May 28, 2026 20:16
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save eruo005-dev/85743066c1773215b259b6595cd8f1ed to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save eruo005-dev/85743066c1773215b259b6595cd8f1ed to your computer and use it in GitHub Desktop.
Download ZIP
runlocalai_llama-3.2-3b-instruct-Q4_K_M-turkishmmlu
Raw
llama-3.2-3b-instruct-Q4_K_M-turkishmmlu.tmp.log
=== runlocalai TurkishMMLU run (generative) ===
model: llama-3.2-3b-instruct (Llama 3.2 3B Instruct)
quant: Q4_K_M
runtime: ollama-0.24
hardware: rtx-3080-16gb-mobile
task: turkishmmlu-generative
n_fewshot: 5
started: 2026-05-28T19:37:57.680Z
finished: 2026-05-28T20:17:00.281Z
wall: 39.0 min
accuracy: 0.11444444444444445
score: 11.4/100
reproduce: pnpm tsx scripts/run-turkish-mmlu.ts --slug llama-3.2-3b-instruct --quant Q4_K_M --runtime ollama-0.24 --hardware rtx-3080-16gb-mobile --endpoint http://localhost:11434/v1 --model-name llama3.2:3b --num-ctx 8192 --python python --notes "English-trained 3B baseline comparison vs Turkish-specialized 8B. Run on RTX 3080 Laptop 16GB, num_ctx=8192. Expected to score near random (20%) or below since model has no Turkish specialization."
=== per subtask ===
Biology 8/100 = 8.0%
Chemistry 6/100 = 6.0%
Geography 7/100 = 7.0%
History 16/100 = 16.0%
Mathematics 11/100 = 11.0%
Philosophy 15/100 = 15.0%
Physics 12/100 = 12.0%
Religion_and_Ethics 6/100 = 6.0%
Turkish_Language_and_Literature 22/100 = 22.0%
=== stdout follows ===
→ Loading TurkishMMLU dataset from HF...
Loading Biology...
5 dev + 100 test
Loading Chemistry...
5 dev + 100 test
Loading Geography...
5 dev + 100 test
Loading History...
5 dev + 100 test
Loading Mathematics...
5 dev + 100 test
Loading Philosophy...
5 dev + 100 test
Loading Physics...
5 dev + 100 test
Loading Religion_and_Ethics...
5 dev + 100 test
Loading Turkish_Language_and_Literature...
5 dev + 100 test
→ Starting evaluation. Model: llama3.2:3b
Endpoint: http://localhost:11434/v1
Few-shot: 5
→ Biology (100 questions)
[ 10/100] correct= 3 ( 30.0%) — 0.39 q/s — ETA 232s
[ 20/100] correct= 3 ( 15.0%) — 0.39 q/s — ETA 205s
[ 30/100] correct= 3 ( 10.0%) — 0.39 q/s — ETA 179s
[ 40/100] correct= 4 ( 10.0%) — 0.39 q/s — ETA 153s
[ 50/100] correct= 7 ( 14.0%) — 0.39 q/s — ETA 128s
[ 60/100] correct= 7 ( 11.7%) — 0.39 q/s — ETA 102s
[ 70/100] correct= 7 ( 10.0%) — 0.39 q/s — ETA 77s
[ 80/100] correct= 7 ( 8.8%) — 0.39 q/s — ETA 51s
[ 90/100] correct= 8 ( 8.9%) — 0.39 q/s — ETA 26s
[100/100] correct= 8 ( 8.0%) — 0.39 q/s — ETA 0s
✓ Biology: 8/100 = 8.0% in 256s
→ Chemistry (100 questions)
[ 10/100] correct= 2 ( 20.0%) — 0.39 q/s — ETA 228s
[ 20/100] correct= 2 ( 10.0%) — 0.39 q/s — ETA 203s
[ 30/100] correct= 2 ( 6.7%) — 0.39 q/s — ETA 178s
[ 40/100] correct= 2 ( 5.0%) — 0.39 q/s — ETA 152s
[ 50/100] correct= 2 ( 4.0%) — 0.39 q/s — ETA 127s
[ 60/100] correct= 2 ( 3.3%) — 0.39 q/s — ETA 102s
[ 70/100] correct= 4 ( 5.7%) — 0.39 q/s — ETA 76s
[ 80/100] correct= 5 ( 6.2%) — 0.39 q/s — ETA 51s
[ 90/100] correct= 6 ( 6.7%) — 0.39 q/s — ETA 26s
[100/100] correct= 6 ( 6.0%) — 0.39 q/s — ETA 0s
✓ Chemistry: 6/100 = 6.0% in 255s
→ Geography (100 questions)
[ 10/100] correct= 1 ( 10.0%) — 0.39 q/s — ETA 230s
[ 20/100] correct= 1 ( 5.0%) — 0.39 q/s — ETA 205s
[ 30/100] correct= 2 ( 6.7%) — 0.39 q/s — ETA 179s
[ 40/100] correct= 3 ( 7.5%) — 0.39 q/s — ETA 154s
[ 50/100] correct= 4 ( 8.0%) — 0.39 q/s — ETA 128s
[ 60/100] correct= 4 ( 6.7%) — 0.39 q/s — ETA 102s
[ 70/100] correct= 7 ( 10.0%) — 0.39 q/s — ETA 77s
[ 80/100] correct= 7 ( 8.8%) — 0.39 q/s — ETA 51s
[ 90/100] correct= 7 ( 7.8%) — 0.39 q/s — ETA 26s
[100/100] correct= 7 ( 7.0%) — 0.39 q/s — ETA 0s
✓ Geography: 7/100 = 7.0% in 256s
→ History (100 questions)
[ 10/100] correct= 1 ( 10.0%) — 0.39 q/s — ETA 231s
[ 20/100] correct= 4 ( 20.0%) — 0.39 q/s — ETA 206s
[ 30/100] correct= 6 ( 20.0%) — 0.39 q/s — ETA 180s
[ 40/100] correct= 6 ( 15.0%) — 0.39 q/s — ETA 153s
[ 50/100] correct= 9 ( 18.0%) — 0.39 q/s — ETA 127s
[ 60/100] correct= 10 ( 16.7%) — 0.39 q/s — ETA 102s
[ 70/100] correct= 13 ( 18.6%) — 0.39 q/s — ETA 76s
[ 80/100] correct= 14 ( 17.5%) — 0.39 q/s — ETA 51s
[ 90/100] correct= 16 ( 17.8%) — 0.39 q/s — ETA 26s
[100/100] correct= 16 ( 16.0%) — 0.39 q/s — ETA 0s
✓ History: 16/100 = 16.0% in 256s
→ Mathematics (100 questions)
[ 10/100] correct= 2 ( 20.0%) — 0.39 q/s — ETA 230s
[ 20/100] correct= 2 ( 10.0%) — 0.39 q/s — ETA 205s
[ 30/100] correct= 4 ( 13.3%) — 0.39 q/s — ETA 179s
[ 40/100] correct= 6 ( 15.0%) — 0.39 q/s — ETA 153s
[ 50/100] correct= 8 ( 16.0%) — 0.39 q/s — ETA 129s
[ 60/100] correct= 8 ( 13.3%) — 0.38 q/s — ETA 104s
[ 70/100] correct= 10 ( 14.3%) — 0.38 q/s — ETA 78s
[ 80/100] correct= 10 ( 12.5%) — 0.38 q/s — ETA 52s
[ 90/100] correct= 11 ( 12.2%) — 0.38 q/s — ETA 26s
[100/100] correct= 11 ( 11.0%) — 0.39 q/s — ETA 0s
✓ Mathematics: 11/100 = 11.0% in 260s
→ Philosophy (100 questions)
[ 10/100] correct= 2 ( 20.0%) — 0.38 q/s — ETA 235s
[ 20/100] correct= 3 ( 15.0%) — 0.38 q/s — ETA 208s
[ 30/100] correct= 3 ( 10.0%) — 0.38 q/s — ETA 182s
[ 40/100] correct= 4 ( 10.0%) — 0.39 q/s — ETA 155s
[ 50/100] correct= 4 ( 8.0%) — 0.39 q/s — ETA 129s
[ 60/100] correct= 6 ( 10.0%) — 0.39 q/s — ETA 103s
[ 70/100] correct= 8 ( 11.4%) — 0.39 q/s — ETA 77s
[ 80/100] correct= 11 ( 13.8%) — 0.39 q/s — ETA 51s
[ 90/100] correct= 13 ( 14.4%) — 0.39 q/s — ETA 26s
[100/100] correct= 15 ( 15.0%) — 0.39 q/s — ETA 0s
✓ Philosophy: 15/100 = 15.0% in 257s
→ Physics (100 questions)
[ 10/100] correct= 0 ( 0.0%) — 0.38 q/s — ETA 236s
[ 20/100] correct= 1 ( 5.0%) — 0.39 q/s — ETA 207s
[ 30/100] correct= 1 ( 3.3%) — 0.39 q/s — ETA 180s
[ 40/100] correct= 2 ( 5.0%) — 0.39 q/s — ETA 154s
[ 50/100] correct= 3 ( 6.0%) — 0.38 q/s — ETA 131s
[ 60/100] correct= 3 ( 5.0%) — 0.38 q/s — ETA 106s
[ 70/100] correct= 6 ( 8.6%) — 0.38 q/s — ETA 79s
[ 80/100] correct= 8 ( 10.0%) — 0.38 q/s — ETA 52s
[ 90/100] correct= 11 ( 12.2%) — 0.38 q/s — ETA 26s
[100/100] correct= 12 ( 12.0%) — 0.38 q/s — ETA 0s
✓ Physics: 12/100 = 12.0% in 260s
→ Religion_and_Ethics (100 questions)
[ 10/100] correct= 0 ( 0.0%) — 0.38 q/s — ETA 234s
[ 20/100] correct= 1 ( 5.0%) — 0.39 q/s — ETA 208s
[ 30/100] correct= 2 ( 6.7%) — 0.39 q/s — ETA 181s
[ 40/100] correct= 2 ( 5.0%) — 0.39 q/s — ETA 155s
[ 50/100] correct= 3 ( 6.0%) — 0.39 q/s — ETA 129s
[ 60/100] correct= 4 ( 6.7%) — 0.39 q/s — ETA 103s
[ 70/100] correct= 5 ( 7.1%) — 0.39 q/s — ETA 78s
[ 80/100] correct= 6 ( 7.5%) — 0.39 q/s — ETA 52s
[ 90/100] correct= 6 ( 6.7%) — 0.39 q/s — ETA 26s
[100/100] correct= 6 ( 6.0%) — 0.39 q/s — ETA 0s
✓ Religion_and_Ethics: 6/100 = 6.0% in 259s
→ Turkish_Language_and_Literature (100 questions)
[ 10/100] correct= 4 ( 40.0%) — 0.39 q/s — ETA 231s
[ 20/100] correct= 7 ( 35.0%) — 0.39 q/s — ETA 205s
[ 30/100] correct= 7 ( 23.3%) — 0.39 q/s — ETA 179s
[ 40/100] correct= 8 ( 20.0%) — 0.39 q/s — ETA 154s
[ 50/100] correct= 8 ( 16.0%) — 0.39 q/s — ETA 128s
[ 60/100] correct= 9 ( 15.0%) — 0.39 q/s — ETA 103s
[ 70/100] correct= 11 ( 15.7%) — 0.39 q/s — ETA 77s
[ 80/100] correct= 15 ( 18.8%) — 0.39 q/s — ETA 52s
[ 90/100] correct= 18 ( 20.0%) — 0.39 q/s — ETA 26s
[100/100] correct= 22 ( 22.0%) — 0.39 q/s — ETA 0s
✓ Turkish_Language_and_Literature: 22/100 = 22.0% in 258s
============================================================
FINAL: 103 / 900 = 11.44%
Wall time: 38.6 min
============================================================
→ Result JSON written to .turkish-mmlu-runs\llama-3.2-3b-instruct-Q4_K_M-result.json
=== stderr follows ===
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment