A latency analysis to decide which hardware signer to use for gno validators. We have a Ledger Nano S Plus on hand. We don't have a YubiHSM 2, so its column is filled in from Yubico's published Ed25519 numbers — good enough to make a decision, but flagged as vendor-reported.
To put either number in context, we also need to know how much overhead gnokms itself adds on top of any backend it talks to. We measure that by running gnokms with the gnokey backend (CPU-only, no hardware) and comparing it against an in-process Ed25519 baseline. That way, when we look at the Ledger number, we can separate gnokms cost from USB cost from on-device cost.
All numbers in ms. Same host throughout: Linux 7.0.3-arch1-2, Ryzen 7 7800X3D, USB Full-Speed HID. Ledger Nano S Plus firmware 1.6.0 running the gnoverse/tm-ledger-validator Ed25519-only validator app (the Tendermint validator app for Ledger — must be sideloaded onto the device).
gnoland supports remote signing: instead of holding the validator private key in-process, it delegates every Sign() call to an external process over a TCP socket or a Unix socket.
gnokms is the sidecar that implements this interface — it accepts sign requests from the validator and forwards them to a pluggable backend (a gnokey keybase, a Ledger device, or in the future a YubiHSM 2).
All latency numbers below were collected with the val-scenarios harness (misc/val-scenarios/, branch chore/benchmark-scenario-tests), which adds a thin valsignerd proxy between the validator and gnokms to record per-phase Sign() latency from live log timestamps.
| signer path | per-sign | dominant cost |
|---|---|---|
| in-process Ed25519 (no gnokms) | <1 ms | actual signing |
| gnokms → gnokey (CPU only) | ~173 ms | bcrypt KDF at cost 12 (~173 ms) |
| gnokms → Ledger Nano S Plus | ~580 ms | on-device Ed25519 (~526 ms) + 50 ms hard-coded sleep per APDU in zondax/ledger-go |
| gnokms → YubiHSM 2 (projected from vendor docs) | ~131 ms | on-device Ed25519 (~130 ms) |
gnokms itself adds ~0.5 ms regardless of backend — measured directly from paired validator/gnokms log timestamps. Everything above that is the backend.
For a ~100 B Tendermint vote, the YubiHSM 2 path would be ~4-5× faster than the Ledger path.
Tendermint's consensus timeouts (timeout_prevote, timeout_precommit) are typically set to 1 s; a signing backend that costs hundreds of milliseconds per call consumes a significant fraction of that budget on every vote.
-
gnokms's own overhead is essentially free: ~0.5 ms per sign (gnokms runs as a local process on the same host; the 0.5 ms covers the TCP loopback round-trip plus gnokms's own serialization: RPC + amino + transport), measured directly as the delta between paired log lines from the validator and the gnokms sidecar. Independent of backend.
-
gnokey-backed signing is essentially bcrypt. ~99.7% of the per-sign cost is the password-KDF step (
bcryptSecurityParameter = 12attm2/pkg/crypto/keys/armor/armor.go:19). No cryptographic op is slow; the cost is a deliberate at-rest key-protection parameter. -
Ledger-backed signing is dominated by on-device Ed25519 (~526 ms), plus a hard-coded 50 ms sleep that fires on every single APDU exchange in
zondax/ledger-go. USB wire time itself is only ~3 ms per APDU. -
Raising the APDU chunk size from 32 to 128 bytes was a ~150 ms win on every sign. The original
32was a macOS HID stability workaround, not a deliberate latency choice; at the new size a typical vote fits in one APDU instead of four, paying the 50 ms host stall once instead of four times. -
Based on vendor numbers, a YubiHSM 2 would sign a ~100 B vote in ~131 ms end-to-end through gnokms, vs ~580 ms for the Ledger.
All measurements come from misc/val-scenarios. The harness runs validators in Docker; each validator has a valsignerd sidecar in front of its signer that records per-phase Sign() latency (count / total / min / max) for every inner Sign call. The chain runs for 100 blocks, then the sidecar prints averages. Per validator over 100 blocks: ~100 prevote samples, ~100 precommit samples, ~25 proposal samples (round-robin among 4 validators).
The only thing that changes between scenarios is what sits behind valsignerd.
Setup: scenario 18 — 4 validators, valsignerd calls Go's crypto/ed25519.Sign() directly in-process. No gnokms, no IPC. 100 blocks. Numbers averaged across all 4 validators' phase samples.
| phase | per-sign |
|---|---|
| proposal | <1 ms |
| prevote | <1 ms |
| precommit | <1 ms |
Raw ed25519.Sign() on this CPU is ~0.02 ms; the rest of the <1 ms is amino encoding and the valsignerd metric path. This is the floor — everything else is overlay on top of this.
Setup: scenario 19 — same 4-validator topology and same 100-block run, but each valsignerd now has a gnokms sidecar (gnokey backend) inserted between it and the priv key. Signing chain: validator → valsignerd → gnokms → gnokey keybase. The point of this scenario is not gnokey itself — it's to isolate gnokms's own RPC/amino/transport cost from the backend cost, so we can apply the same constant to the Ledger numbers below.
Note: gnokey's standard mnemonic-derived keys are secp256k1, but its ImportPrivKey API accepts any algorithm, and val-scenarios uses it to load the validator's Ed25519 key directly from priv_validator_key.json. So this measurement is Ed25519 signing, the same algorithm as the Ledger path below.
Paired log lines from the same sign event:
val1-1 | INFO signature received from gnokms duration: 173.18 ms
val1-gnokms-1 | INFO signature produced by gnokey duration: 172.77 ms
Decomposed:
| component | time | how known |
|---|---|---|
| validator → gnokms RPC + amino + transport | ~0.5 ms | paired log delta: val1 "signature received from gnokms" − gnokms "signature produced by gnokey" |
| bcrypt at cost 12 (gnokey side) | ~173 ms | gnokms-side log; matches independent bcrypt benchmark (~174 ms/op) |
| in-process Ed25519 | <1 ms | scenario 18 |
| total | ~173 ms | val1-side log line |
The takeaway for the rest of this gist: gnokms itself costs ~0.5 ms — the RPC layer is essentially free, and ~99.7% of gnokey-backed signing is bcrypt. We'll apply this same constant to the Ledger total below to isolate the Ledger-specific cost.
Setup: scenario 20 — 1 ledger-backed validator (val1) + 3 gnokey-backed validators (val2-val4) for quorum. val1's chain is validator → valsignerd → host gnokms (ledger) → Ledger Nano S Plus over USB-HID. gnokms runs directly on the host (CGO + USB) because USB-HID inside Docker is unreliable on macOS; the in-Docker valsignerd reaches it via host.docker.internal. 100 blocks. First sign after device wake is a ~4000 ms outlier (on-device user confirmation request) and is excluded.
Per-phase steady-state averages are around 527-584 ms. Headline ~580 ms per sign.
The question is: what's in those 580 ms?
The Ledger communicates over USB using APDU (Application Protocol Data Unit) frames — short command/response packets exchanged one at a time. Each APDU exchange carries a fixed overhead, so the number of exchanges for a single sign operation directly determines USB wire cost.
The gnokms ledger code on branch chore/ledger-benchmarks was instrumented to log per-Exchange duration, and a parallel USB trace was captured with usbmon + Wireshark.
Raw capture of a single getVersion APDU (USB Full-Speed, Ledger Nano S Plus):
# │ Time │ Direction │ Size │ URB
───┼────────────┼───────────┼───────┼────────────────────────────────────────────────
26 │ 229.118941 │ host→dev │ 128 B │ INTERRUPT OUT submit (carries 64 B HID payload)
27 │ 229.119123 │ dev→host │ 64 B │ INTERRUPT OUT complete (kernel ack)
28 │ 229.122119 │ dev→host │ 128 B │ INTERRUPT IN complete (response from device)
29 │ 229.122158 │ host→dev │ 64 B │ INTERRUPT IN submit (next poll queued)
30 │ 229.172765 │ host→dev │ 128 B │ INTERRUPT OUT submit (next cycle begins)
From this trace:
- OUT submit → OUT complete: 0.18 ms. Kernel acks the host's write into the USB stack; no device involvement yet.
- OUT complete → IN complete: 3.00 ms. Device receives the OUT, parses the APDU, runs the handler, sends the IN response, kernel completes the IN URB. This is the inherent cadence of USB Full-Speed interrupt transfers — the device's HID descriptor sets
bInterval = 1ms, so each round-trip costs 1-3 polling slots. This is not a USB wake-up; the device is already enumerated and being polled continuously. It's the wire-protocol floor, and there is no software trick that makes it faster. - Wire round-trip total: 3.18 ms (OUT submit → IN complete).
- IN complete → next OUT submit: 50.65 ms. USB bus is idle here, the kernel has nothing pending — every millisecond of this is user-space.
That 50 ms gap reproduces on every Exchange. The source is in zondax/ledger-go/ledger_hid.go:204 (zondax/ledger-go is a third-party Go library that gnokms uses internally to talk to the Ledger device; the sleep is not in gno code and cannot be patched without forking the library.):
func (ledger *LedgerDeviceHID) drainRead() {
// Allow time for late packet arrivals (When main program doesn't read enough packets)
<-time.After(50 * time.Millisecond)
for {
select {
case <-ledger.readChannel:
default:
return
}
}
}drainRead() is called unconditionally at the top of every Exchange() call. The comment justifies it as a guard against straggler packets from the previous response, but the time.After fires whether anything is pending or not. This single <-time.After(50 * time.Millisecond) is the entire "host stall."
Paired log lines for the same sign event:
val1-1 | INFO signature received from gnokms duration: 580 ms (steady-state)
gnokms | INFO signature received from ledger duration: ~579.5 ms
For a one-chunk sign (vote payload ≤128 B, sent in a single HID frame):
| component | time | source |
|---|---|---|
| gnokms RPC + amino + transport | ~0.5 ms | paired log delta: val1 "signature received from gnokms" − gnokms "signature received from ledger" |
host-side sleep before write (drainRead) |
~50 ms | zondax/ledger-go/ledger_hid.go:204, confirmed in wireshark gap |
| USB wire round-trip | ~3 ms | wireshark, OUT submit → IN complete |
| on-device Ed25519 sign | ~526 ms | residual: 580 − 0.5 − 50 − 3 |
| per sign | ~580 ms |
The on-device sign alone (~526 ms) is more than 4× the entire YubiHSM 2 sign time for the same payload. That's the headline cost.
The original implementation set ledgerMessageChunkSize = 32 with the comment needed for macOS HID IO issues (commit 8a4d22989, contribs/gnokms/internal/ledger/ledger_signer.go). 32 bytes was a stability workaround for macOS HID write quirks, not a deliberate latency choice. It has since been raised to 128 to optimize this benchmark, and because we can consider that future validators will not be on macOS.
Because each APDU exchange carries a fixed ~50 ms penalty from drainRead(), reducing the number of exchanges directly reduces total cost. Raising the APDU chunk size from 32 to 128 bytes cut the exchanges needed for a typical ~100 B Tendermint vote from 4 to 1, saving ~150 ms per sign call. On-device Ed25519 signing time (~526 ms) was unaffected:
| chunk size | exchanges for ~100 B vote | predicted total | observed |
|---|---|---|---|
| 32 (original, macOS stability) | 3-4 (depends on payload) | N × 53 + 526 ≈ 685-738 ms | ~670-730 ms (confirmed by deliberate A/B on this branch) |
| 128 (current) | 1 | 1 × 53 + 526 ≈ 579 ms | ~580 ms |
The ~150 ms improvement is purely from no longer paying the drainRead stall multiple times. The on-device cost did not change.
We don't have a YubiHSM 2 to benchmark, so this is built from Yubico's published Ed25519 latencies for an otherwise-unoccupied device (source):
| payload | YubiHSM 2 sign latency (vendor) |
|---|---|
| 32 B | ~105 ms |
| 64 B | ~121 ms |
| 128 B | ~137 ms |
| 256 B | ~168 ms |
| 512 B | ~229 ms |
| 1024 B | ~353 ms |
A typical Tendermint vote payload sits between 64 and 128 B → call it ~130 ms on-device. Adding our measured gnokms overhead:
| component | time |
|---|---|
| gnokms RPC + amino + transport | ~0.5 ms |
| YubiHSM 2 on-device Ed25519, ~100 B payload | ~130 ms |
| transport (PKCS#11 / yubihsm-shell over USB) | not separately published — bundled in vendor number |
| per sign (projected) | ~131 ms |
Caveats:
- Yubico measures on an "otherwise unoccupied" device. Under concurrent load these numbers will be higher.
- The gnokms YubiHSM 2 backend doesn't exist yet — implementing it is part of what this evaluation is meant to inform.
- The transport cost (PKCS#11 module / yubihsm-shell over USB) is folded into Yubico's number, but the gnokms-side adapter would add a small amount on top.
Even accounting for these, the ~4-5× gap vs the Ledger is large enough that benchmarking a real YubiHSM 2 isn't expected to flip the decision.
A YubiHSM 2, based on vendor numbers, would land around ~131 ms end-to-end through gnokms — about 4-5× faster than the Ledger Nano S Plus (this figure comes from the YubiHSM 2 datasheet's Ed25519 signing specification; it has not yet been validated against real hardware in the gnokms signing path).
The next step is to acquire a YubiHSM 2, implement a gnokms backend for it, and re-run scenario 19/20-style benchmarks against the real device to confirm or correct the vendor-derived projection.
Two branches are involved: chore/benchmark-scenario-tests contains the val-scenarios harness (scenarios 18–21, the valsignerd sidecar, and this document's measurement tooling); chore/ledger-benchmarks contains the gnokms ledger backend with the raised chunk size + operation duration logs. The image builds pull from chore/ledger-benchmarks so they include both.
# 1. Get the branches
git clone https://github.com/D4ryl00/gno
cd gno
# 2. Baseline + gnokey scenarios (Docker only)
git checkout chore/benchmark-scenario-tests
cd misc/val-scenarios
GH_USER=d4ryl00 GH_REPO=gno GH_BRANCH=chore/ledger-benchmarks VALSIGNER_GNO_ROOT=../.. make build-images
GH_USER=d4ryl00 GH_REPO=gno GH_BRANCH=chore/ledger-benchmarks GNO_ROOT=../.. make scenario-18 # local Ed25519
GH_USER=d4ryl00 GH_REPO=gno GH_BRANCH=chore/ledger-benchmarks GNO_ROOT=../.. make scenario-19 # gnokms + gnokey
# 3. Ledger scenario (requires a Ledger Nano S Plus with the
# gnoverse/tm-ledger-validator app installed, and Ledger Live closed)
make build-gnokms-ledger # builds host-side gnokms from chore/ledger-benchmarks
GH_USER=d4ryl00 GH_REPO=gno GH_BRANCH=chore/ledger-benchmarks GNO_ROOT=../.. make scenario-20 # 1 ledger validator + 3 gnokey validators
# 4. USB capture (Linux)
sudo modprobe usbmon
sudo wireshark # interface: usbmonN, filter usb.idVendor == 0x2c97
# 5. Measure gnokms's own overhead from paired log lines
# The gnokms layer's cost is measured directly as the delta between two
# paired log lines emitted around the same sign call.
# Validator-side (/tmp/gno-val-tests/scenario-19/logs/val1.log): time from "send to gnokms" to "got signature back"
cat /tmp/gno-val-tests/scenario-19/logs/val1.log | grep "signature received from gnokms"
# val1-1 | INFO signature received from gnokms {"duration": "173.18ms"}
# gnokms-side (/tmp/gno-val-tests/scenario-19/logs/val1-gnokms.log): time gnokms itself spent calling the backend
cat /tmp/gno-val-tests/scenario-19/logs/val1-gnokms.log | grep "signature produced by gnokey"
# val1-gnokms-1 | INFO signature produced by gnokey {"duration": "172.77ms"}
# Delta = gnokms RPC + amino + transport overhead per sign.
# scenario 19 (docker-internal network): ~0.4 ms
# scenario 20 (host.docker.internal): ~0.5 ms
# The same technique works for scenario 20, using the gnokms ledger log line
# "signature received from ledger".valsignerd exposes per-phase metrics on each validator's signer-control port (validator RPC port + 1); print_all_signer_metrics at the end of each scenario prints them. See misc/val-scenarios/README.md for the full harness API.