Skip to content

Instantly share code, notes, and snippets.

@slewsys

slewsys/benchmark.md

Last active September 2, 2026 14:00
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save slewsys/da27a118587b6a50abe1218c55f215a0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save slewsys/da27a118587b6a50abe1218c55f215a0 to your computer and use it in GitHub Desktop.
Download ZIP
llama.cpp benchmarks on internal and external RTX 3090s.
Raw
benchmark.md

GIGABYTE AORUS RTX 3090 Gaming Box w/ Thunderbolt 3 (PCIe 3.0 x4)

The following benchmark is run on an HP Z8 G4 with dual Xeon Gold 6230 CPUs, 384 GB DDR4 2666 MHz DIMMs and AORUS RTX 3090 eGPU connected to an HP Thunderbolt 3 card.

env CUDA_VISIBLE_DEVICES=0 llama-bench                \
    --model "${LLAMA_CACHE}/gpt-oss-120b-mxfp4.gguf"  \
    --threads 1                                       \
    --flash-attn 1                                    \
    --batch-size 4096                                 \
    --ubatch-size 2048,4096                           \
    --n-gpu-layers 12                                 \
    --n-prompt 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:

  • Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
model size params backend threads n_batch n_ubatch fa test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp2048 140.46 ± 0.95
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp8192 141.23 ± 0.46
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp16384 140.11 ± 0.65
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp32768 137.44 ± 0.62
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 tg128 2.27 ± 0.00
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp2048 141.00 ± 0.78
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp8192 225.28 ± 1.11
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp16384 224.42 ± 0.89
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp32768 219.08 ± 0.83
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 tg128 3.15 ± 0.01

build: ee39cc8 (6832)

NVIDIA GeForce RTX 3090 Founder's Edition in PCIe 3.0 x16 Slot

The following benchmark is run on an HP Z8 G4 with dual Xeon Gold 6230 CPUs, 384 GB DDR4 2666 MHz DIMMs and internal NVIDIA Founder's Edition RTX 3090 GPU in PCIe 3.0 x16 slot.

env CUDA_VISIBLE_DEVICES=1 llama-bench                \
    --model "${LLAMA_CACHE}/gpt-oss-120b-mxfp4.gguf"  \
    --threads 1                                       \
    --flash-attn 1                                    \
    --batch-size 4096                                 \
    --ubatch-size 2048,4096                           \
    --n-gpu-layers 12                                 \
    --n-prompt 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:

  • Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
model size params backend threads n_batch n_ubatch fa test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp2048 303.84 ± 4.91
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp8192 308.73 ± 1.16
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp16384 267.18 ± 29.29
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp32768 290.99 ± 2.54
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 tg128 2.33 ± 0.00
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp2048 303.17 ± 1.68
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp8192 454.81 ± 3.83
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp16384 382.19 ± 5.96
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp32768 429.01 ± 13.86
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 tg128 3.23 ± 0.02

build: ee39cc8 (6832)

2x RTX 3090 (AORUS + Founder's Edition/PCIe 3.0)

The following benchmark is run on an HP Z8 G4 with dual Xeon Gold 6230 CPUs, 384 GB DDR4 2666 MHz DIMMs, internal NVIDIA Founder's Edition GPU and external AORUS eGPU in parallel. The internal GPU (Device 1) is the "main" GPU. The results are roughly the sum of the results of the two individual GPUs.

env CUDA_VISIBLE_DEVICES=1,0 llama-bench              \
    --model "${LLAMA_CACHE}/gpt-oss-120b-mxfp4.gguf"  \
    --threads 1                                       \
    --flash-attn 1                                    \
    --batch-size 4096                                 \
    --ubatch-size 2048,4096                           \
    --n-prompt 2048,8192,16384,32768                  \
    --n-gpu-layers 24                                 \
    --tensor-split 50/50

CUDA_VISIBLE_DEVICES='1,0' ./run-ggml-gpt-oss-120b-mxfp4.sh --bench ggml_cuda_init: found 2 CUDA devices:

  • Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  • Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
model size params backend threads n_batch n_ubatch fa test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp2048 544.36 ± 15.87
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp8192 561.21 ± 17.32
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp16384 561.54 ± 5.93
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp32768 524.77 ± 2.54
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 tg128 7.09 ± 0.04
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp2048 558.60 ± 8.89
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp8192 806.74 ± 3.68
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp16384 800.37 ± 0.46
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp32768 759.63 ± 1.93
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 tg128 5.20 ± 0.02

build: 2d8b857e0 (7761)

2x RTX 3090 (AORUS + Founder's Edition/PCIe 3.0)

The following benchmark is run on an HP Z8 G4 with dual Xeon Gold 6230 CPUs, 384 GB DDR4 2666 MHz DIMMs, internal NVIDIA Founder's Edition GPU and external AORUS eGPU in parallel. The eGPU (Device 0) is the "main" GPU. The results are roughly the average of the results of the two individual GPUs.

env CUDA_VISIBLE_DEVICES=0,1 llama-bench              \
    --model "${LLAMA_CACHE}/gpt-oss-120b-mxfp4.gguf"  \
    --threads 1                                       \
    --flash-attn 1                                    \
    --batch-size 4096                                 \
    --ubatch-size 2048,4096                           \
    --n-prompt 2048,8192,16384,32768                  \
    --n-gpu-layers 24                                 \
    --tensor-split 50/50

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:

  • Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  • Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
model size params backend threads n_batch n_ubatch fa test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp2048 223.72 ± 2.67
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp8192 222.47 ± 0.43
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp16384 220.49 ± 0.66
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp32768 214.26 ± 0.60
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 tg128 5.31 ± 0.04
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp2048 222.94 ± 1.89
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp8192 361.70 ± 1.82
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp16384 355.61 ± 0.27
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 pp32768 343.64 ± 0.69
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 4096 1 tg128 5.29 ± 0.02

build: ee39cc8 (6832)

GIGABYTE AORUS RTX 3090 Gaming Box w/ Thunderbolt 3 (PCIe 4.0 x4)

The following benchmark is run on an HP Z8 G4 with dual Xeon Gold 6230 CPUs, 384 GB DDR4 2666 MHz DIMMs and AORUS RTX 3090 eGPU connected to a Dell Inspiron 7420 Plus laptop with Thunderbolt 4 port and PCIe 4.0 bus. The results are so poor that the benchmark was stopped before completing.

env CUDA_VISIBLE_DEVICES=1 llama-bench                \
    --model "${LLAMA_CACHE}/gpt-oss-120b-mxfp4.gguf"  \
    --threads 1                                       \
    --flash-attn 1                                    \
    --batch-size 4096                                 \
    --ubatch-size 2048,4096                           \
    --n-gpu-layers 12                                 \
    --n-prompt 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:

  • Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
model size params backend threads n_batch n_ubatch fa test t/s
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp2048 97.13 ± 8.67
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp8192 104.17 ± 0.60
gpt-oss 120B MXFP4 MoE 59.02 GiB 116.83 B CUDA,BLAS 1 4096 2048 1 pp16384 103.76 ± 0.67
...

build: fb5f3b0 (6533)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment