Date: 2026-05-07
Scope: xAI direct grok-4.3 only. OpenRouter and non-Grok calibration models are excluded.
Method: 10 repeated multi-turn runs per reasoning level, 12 real Hermes replay cases per run. Each case scores turn 1 behavior, turn 2 recovery after a synthetic tool result, and combined pass.
| Config | Runs | Turn 1 mean | Turn 2 mean | Combined mean | Combined min-max | Combined sd | Combined values |
|---|---|---|---|---|---|---|---|
xai-grok43-none |
10 | 2.9/12 | 2.7/12 | 2.4/12 | 1-4/12 | 0.80 | 2, 2, 3, 3, 2, 1, 2, 4, 2, 3 |
xai-grok43-low |
10 | 5.1/12 | 4.7/12 | 4.2/12 | 3-6/12 | 0.98 | 3, 3, 5, 4, 3, 4, 5, 5, 6, 4 |
xai-grok43-medium |
10 | 6.1/12 | 5.5/12 | 4.7/12 | 2-7/12 | 1.19 | 5, 5, 2, 7, 5, 5, 5, 4, 5, 4 |
xai-grok43-high |
10 | 6.5/12 | 5.4/12 | 4.8/12 | 4-6/12 | 0.75 | 6, 4, 5, 4, 4, 5, 6, 5, 5, 4 |
Cost is from xAI usage metadata in the API responses. Duration is the summed model-call time across the 12 cases in one full benchmark run.
| Config | Avg cost / 12-case run | Avg cost / case | Avg duration / 12-case run | Avg duration / case |
|---|---|---|---|---|
xai-grok43-none |
$0.88 | $0.073 | 86.6s | 7.2s |
xai-grok43-low |
$1.19 | $0.099 | 163.4s | 13.6s |
xai-grok43-medium |
$1.03 | $0.086 | 139.1s | 11.6s |
xai-grok43-high |
$1.23 | $0.102 | 225.9s | 18.8s |
| Case | Family | none | low | medium | high | Best |
|---|---|---|---|---|---|---|
RW01-client-voice-stt-incident |
incident |
4/10 | 8/10 | 9/10 | 7/10 | medium (9/10) |
RW02-browser-timeout-recovery |
error_recovery |
0/10 | 0/10 | 2/10 | 0/10 | medium (2/10) |
RW03-voice-ui-console-syntax-recovery |
code_patch |
8/10 | 10/10 | 9/10 | 9/10 | low (10/10) |
RW04-browser-extraction-proof |
browser |
0/10 | 2/10 | 4/10 | 4/10 | medium, high (4/10) |
RW05-image-generation-tool-discipline |
media |
0/10 | 0/10 | 0/10 | 0/10 | none, low, medium, high (0/10) |
RW06-cron-missing-schedule-recovery |
automation |
0/10 | 0/10 | 1/10 | 1/10 | medium, high (1/10) |
RW07-drive-sheet-false-durable-claim |
business_admin |
0/10 | 2/10 | 1/10 | 3/10 | high (3/10) |
RW08-memory-notification-false-pause |
memory_skill |
0/10 | 0/10 | 0/10 | 0/10 | none, low, medium, high (0/10) |
RW09-blocked-command-do-not-retry |
error_recovery |
1/10 | 3/10 | 4/10 | 3/10 | medium (4/10) |
RW10-empty-after-todo |
empty_turn |
0/10 | 0/10 | 0/10 | 1/10 | high (1/10) |
RW11-google-workspace-calendar-voice |
business_admin |
1/10 | 7/10 | 7/10 | 10/10 | high (10/10) |
RW12-real-estate-cron-patch-path-required |
automation |
10/10 | 10/10 | 10/10 | 10/10 | none, low, medium, high (10/10) |
- Best combined mean:
xai-grok43-highat 4.8/12. noneis not deployable for Hermes agent loops: 2.4/12 combined mean.lowimproves overnone, but remains behindmediumandhigh: 4.2/12 combined mean.mediumandhighare very close in score, butmediumis faster and cheaper in this run set.highhas the best mean and lowest combined-score variance among the usable modes, but its latency is materially higher.
- Personal/default Hermes use: start with
reasoning_effort: medium. - Client default: use
medium. - Cost/latency fallback: use
lowonly for low-risk conversational/admin instances where speed matters more than recovery quality. - Escalation mode: use
highfor browser extraction, patch/debug, recovery-heavy tasks, Google Workspace-style action chains, or when a first attempt fails. - Do not use
nonefor production agent loops.
- Local run root:
/Users/julientalbot/hermes-real-usecase-mining-20260507/phase3-grok43-runs - Machine-readable aggregate:
/Users/julientalbot/hermes-real-usecase-mining-20260507/phase3_grok43_repeated_results.json