README.md

Qwen3.8 27B endpoint performance comparison

Technical summary

Port 8081 is the stronger endpoint for response generation, while cold-prompt latency is effectively tied. In a matched 256-token streaming workload, port 8081 produced tokens at a median 27.91 tokens/s, versus 16.16 tokens/s on port 1234: a 72.7% advantage. Median end-to-end latency fell from 16.33 seconds to 9.78 seconds, a 40.1% reduction.

For a cold 4,081-token prompt, median time to first token (TTFT) was 7.13 seconds on port 1234 and 7.21 seconds on port 8081. The 82 ms difference is only 1.2% and is smaller than the observed run-to-run variation, so this benchmark does not establish a meaningful cold-prefill winner.

Both endpoints cache growing conversation prefixes effectively. Across controlled conversations, the prompt grew from 448 to 3,153 tokens while median TTFT remained close to 1.2 seconds on both servers. At the final cached turn, TTFT was essentially identical, but port 8081 still completed the 32-token response 25.2% sooner because its token generation remained faster.

The practical conclusion is straightforward: use port 8081 for interactive chat and response-heavy work. Port 1234 offers no demonstrated cold-prefill advantage large enough to offset its substantially slower sustained generation.

Port 8081 is 73% faster at sustained token generation

The response-generation test used a 99-token chat prompt and forced every response to the 256-token limit. Five measured trials followed warm-up requests. Endpoint order alternated by trial, prompt caching was disabled, temperature was zero, and each prompt began with a unique nonce.

Metric	Port 1234	Port 8081	Port 8081 difference
Output rate, median	16.16 tok/s	27.91 tok/s	+72.7%
Output rate, mean	16.11 tok/s	30.16 tok/s	+87.2%
Output-rate range	15.77–16.46 tok/s	25.35–43.05 tok/s	—
End-to-end rate, median	15.68 tok/s	26.18 tok/s	+67.0%
Total latency, median	16.33 s	9.78 s	−40.1%
Total latency, p90	16.71 s	10.73 s	−35.8%
TTFT, median	0.589 s	0.621 s	+5.5% (slower)

Port 1234 was slower but exceptionally stable: its output-rate standard deviation was only 0.27 tok/s. Port 8081 was consistently faster, though more variable, with a 7.39 tok/s standard deviation and one 43.05 tok/s trial. The conservative comparison is therefore the median, not the mean. Even the slowest 8081 trial, 25.35 tok/s, exceeded the fastest 1234 trial, 16.46 tok/s by 54%.

The 32 ms median TTFT advantage for port 1234 is real in the recorded sample but small. For responses longer than roughly one or two tokens, port 8081's generation advantage quickly dominates that initial difference.

Generation trial detail

Trial	1234 output tok/s	8081 output tok/s	1234 total s	8081 total s
1	15.77	25.35	16.84	10.77
2	16.46	25.38	16.09	10.66
3	16.16	43.05	16.29	6.54
4	15.94	29.09	16.52	9.39
5	16.20	27.91	16.33	9.78

Cold 4k-token prompt performance is a statistical tie

The cold-prefill test used a 4,081-token prompt followed by a 32-token response. A unique nonce was placed at the beginning of every prompt. This detail matters: an initial discarded run placed the nonce at the end, allowing port 1234 to reuse nearly the entire common prefix despite cache_prompt:false. Moving the nonce to the front eliminated that contamination.

Metric	Port 1234	Port 8081	Difference
TTFT, median	7.131 s	7.213 s	8081 +1.2%
TTFT, mean	6.911 s	6.988 s	8081 +1.1%
TTFT standard deviation	0.689 s	0.794 s	—
TTFT range	6.130–7.550 s	6.084–7.802 s	—
Total latency, median	8.592 s	8.191 s	8081 −4.7%
Post-first-token output rate, median	21.35 tok/s	31.67 tok/s	8081 +48.4%

TTFT is the relevant cold-prefill metric because it measures the client-observed interval from request submission through scheduling, template processing, prompt evaluation, and production of the first streamed token. The median TTFT gap is only 0.082 seconds, while within-endpoint standard deviations are 0.69–0.79 seconds. With five trials per endpoint and overlapping ranges, declaring either server faster at cold prefill would overstate the evidence.

Port 8081's lower median total latency comes from faster generation of the 32 output tokens, not demonstrably faster prompt evaluation.

Cold-prefill trial detail

Trial	1234 TTFT s	8081 TTFT s	1234 total s	8081 total s
1	6.130	6.084	7.567	7.063
2	7.550	6.218	9.029	7.200
3	7.516	7.620	8.968	8.848
4	7.131	7.802	8.592	8.778
5	6.227	7.213	7.673	8.191

Prompt caching keeps TTFT flat as conversation history grows 7×

The cache test simulated a conversation that grew over six turns. Three independent conversation replicates were run per endpoint. Each request retained the prior history and added a controlled user/assistant exchange, producing byte-for-byte comparable histories across endpoints. Prompt caching was enabled. The first system message carried a replicate-specific nonce to prevent reuse across independent conversations.

The prompt increased from 448 tokens on turn 1 to 3,153 tokens on turn 6, a 7.0× increase. Median TTFT did not grow with the full history size, demonstrating that both servers reused the cached prefix and primarily evaluated the newly appended portion.

Turn	Prompt tokens	1234 median TTFT	8081 median TTFT	1234 median total	8081 median total
1	448	1.275 s	1.243 s	2.668 s	2.300 s
2	989	1.172 s	1.148 s	3.476 s	2.286 s
3	1,530	1.191 s	1.230 s	2.924 s	2.183 s
4	2,071	1.183 s	1.185 s	3.267 s	2.240 s
5	2,612	1.197 s	1.188 s	3.303 s	2.249 s
6	3,153	1.192 s	1.193 s	3.021 s	2.260 s

From turn 1 to turn 6, median TTFT changed by −6.5% on port 1234 and −4.0% on port 8081 despite the much larger prompt. At turn 6, the endpoints differed by less than 1 ms in median TTFT. Port 8081 nevertheless returned the complete 32-token response 0.761 seconds sooner, a 25.2% latency reduction, because its median post-first-token rate was 29.16 tok/s versus 16.94 tok/s.

This scenario is deliberately controlled rather than fully naturalistic. Fixed assistant-history text was used so both endpoints received identical prompts; the generated benchmark response was not inserted into the next turn. That preserves experimental comparability but slightly understates the maximum cache reuse possible when a real client sends the model's exact previous response back in the next request.

Scope and metric definitions

Endpoints

• Port 1234: LM Studio local server, model ID qwen3.8-27b, Qwen35 architecture, Q4_K_M, 262,144-token loaded context. Its native API identified the runtime as llama.cpp-linux-arm64-nvidia-cuda13 version 2.28.2 during preliminary checks.
• Port 8081: locally compiled upstream llama.cpp commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd, main Q4_K_M model plus Q4_0 MTP draft, --spec-default --spec-type draft-mtp, 262,144-token context.
• Hardware: NVIDIA GB10, compute capability 12.1, driver 580.82.09. Both endpoints ran on the same machine and shared GPU.

Metrics

• TTFT: client-side monotonic time from sending the HTTP request until the first non-empty streamed content or reasoning_content delta.
• Output tokens/s: (completion tokens − 1) ÷ (stream completion time − first-token time). Subtracting one token avoids counting the first token in both TTFT and the generation interval.
• End-to-end tokens/s: completion tokens divided by total client-observed request time.
• Total latency: request submission through receipt of the completed SSE stream.
• p90: linearly interpolated 90th percentile across five measured trials. With this small sample it is descriptive, not a tail-latency guarantee.

Methodology and reproducibility

1. Both servers were already loaded before measurement.
2. A short warm-up request was issued to each endpoint and excluded.
3. Requests used the OpenAI-compatible streaming chat-completions API.
4. Temperature was fixed at zero.
5. Every measured response reached the requested token limit, giving equal completion-token denominators.
6. Endpoint order alternated by trial to reduce systematic order and thermal bias.
7. The cold tests placed unique nonces at the beginning of prompts and requested cache_prompt:false.
8. The conversation test intentionally enabled caching and grew the same controlled history on both endpoints.
9. All primary timing comparisons use the same client clock rather than mixing vendor-specific server counters.

Reproduction scripts:

• benchmark_endpoints.py — five-trial cold-prefill and sustained-generation tests.
• benchmark_conversation_cache.py — three replicates of six-turn growing conversations.

Run them from this directory:

python3 benchmark_endpoints.py
python3 benchmark_conversation_cache.py

Limitations, uncertainty, and robustness checks

• Small samples: five cold/generation trials and three cached-conversation replicates are enough to reject the earlier single-run narrative, but not enough for production-grade percentile estimates or formal equivalence testing.
• Shared hardware state: both models were resident on the same GB10. Requests were sequential, so they did not compete for active compute, but shared memory residency and background system activity can affect absolute numbers.
• Different server builds and configuration surfaces: the comparison measures the endpoints as deployed, not isolated compiler effects. Port 8081's exact command is known; port 1234's full launch flags are not exposed by its API.
• Speculative decoding variability: port 8081's generation rate varied substantially, consistent with content-dependent draft acceptance. The median is therefore the preferred operational estimate.
• Reasoning tokens: the model emitted reasoning content, and those tokens are counted as completion tokens because they consume decode work and are streamed to the client.
• Streaming chunk timing: TCP buffering and server flush behavior can slightly shift measured first-token arrival. Both endpoints were accessed over loopback with the same client, minimizing but not eliminating this effect.
• No concurrency test: results apply to one request at a time. Throughput under multiple simultaneous users may rank differently because batching and KV-cache pressure change.
• No very-long-context sweep: the cold test covers approximately 4k input tokens, while the cache test grows to approximately 3k. It does not characterize 32k–262k contexts.

Validation assessment: ready to share with caveats

The headline generation result is robust within the observed sample: every port 8081 generation trial exceeded every port 1234 trial. The cold-prefill result should be treated as a tie because differences are small relative to variation. The prompt-cache conclusion is strongly supported directionally by flat TTFT across a sevenfold history increase, but exact cache-hit counts were not exposed through the common API.

Recommended next steps

1. Prefer port 8081 for normal interactive use. Its median sustained output rate is roughly 1.73× port 1234's, with materially lower complete-response latency.
2. Retain prompt caching for conversations. Both endpoints keep TTFT near 1.2 seconds as the controlled history grows to 3,153 tokens.
3. Benchmark concurrency before choosing a multi-user server. Repeat at 2, 4, and 8 simultaneous requests and record aggregate throughput, per-request p50/p95 TTFT, and failure rate.
4. Repeat the generation test with several content types. Code, prose, JSON, and tool-call outputs can produce different MTP acceptance rates and may explain port 8081's variability.
5. Capture native speculative counters in a follow-up. Acceptance rate, drafted tokens, and accepted tokens would help attribute the generation gap rather than only describe it.

Further questions

• Does port 8081 retain its advantage with concurrency and continuous batching?
• How does the comparison change at 8k, 32k, and 128k cold contexts?
• Which server has better p95 TTFT over 50–100 trials?
• How much of port 8081's generation variance is explained by MTP acceptance rate?

benchmark_conversation_cache.py

#!/usr/bin/env python3
"""Controlled growing-conversation benchmark for prompt-prefix caching."""

import json
import statistics
import time
import urllib.request


ENDPOINTS = {
    "1234": "http://127.0.0.1:1234/v1/chat/completions",
    "8081": "http://127.0.0.1:8081/llama/v1/chat/completions",
}

TURN_TEXT = (
    "Discuss memory bandwidth, quantization, KV-cache growth, batching, CUDA kernels, "
    "speculative decoding acceptance, synchronization, and latency tradeoffs. " * 14
)

FIXED_ASSISTANT = (
    "The controlled history uses a fixed assistant turn so both endpoints receive byte-for-byte "
    "equivalent histories. This is a benchmark fixture rather than model-generated evidence. " * 5
)


def request(endpoint, messages):
    body = {
        "model": "qwen3.8-27b",
        "messages": messages,
        "max_tokens": 32,
        "temperature": 0,
        "stream": True,
        "stream_options": {"include_usage": True},
        "cache_prompt": True,
    }
    req = urllib.request.Request(
        ENDPOINTS[endpoint],
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    started = time.perf_counter()
    first = None
    usage = None
    with urllib.request.urlopen(req, timeout=180) as response:
        for raw in response:
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            event = json.loads(payload)
            if event.get("usage"):
                usage = event["usage"]
            for choice in event.get("choices", []):
                delta = choice.get("delta", {})
                if delta.get("content") or delta.get("reasoning_content"):
                    if first is None:
                        first = time.perf_counter()
    ended = time.perf_counter()
    if first is None or usage is None:
        raise RuntimeError(f"incomplete response from {endpoint}")
    return {
        "prompt_tokens": usage["prompt_tokens"],
        "completion_tokens": usage["completion_tokens"],
        "ttft_s": first - started,
        "total_s": ended - started,
        "output_tokens_per_s": (usage["completion_tokens"] - 1) / (ended - first),
    }


def main():
    records = []
    for replicate in range(1, 4):
        endpoint_order = ["1234", "8081"] if replicate % 2 else ["8081", "1234"]
        for endpoint in endpoint_order:
            messages = [{
                "role": "system",
                "content": f"Controlled cache benchmark conversation {replicate}-{endpoint}.",
            }]
            for turn in range(1, 7):
                messages.append({
                    "role": "user",
                    "content": f"Turn {turn}. {TURN_TEXT}",
                })
                result = request(endpoint, messages)
                result.update({"endpoint": endpoint, "replicate": replicate, "turn": turn})
                records.append(result)
                print(json.dumps({"progress": result}), flush=True)
                messages.append({"role": "assistant", "content": FIXED_ASSISTANT})

    summary = {}
    for endpoint in ENDPOINTS:
        summary[endpoint] = {}
        for turn in range(1, 7):
            rows = [r for r in records if r["endpoint"] == endpoint and r["turn"] == turn]
            summary[endpoint][str(turn)] = {
                "prompt_tokens": [r["prompt_tokens"] for r in rows],
                "ttft_median_s": statistics.median(r["ttft_s"] for r in rows),
                "ttft_mean_s": statistics.mean(r["ttft_s"] for r in rows),
                "total_median_s": statistics.median(r["total_s"] for r in rows),
                "output_tps_median": statistics.median(r["output_tokens_per_s"] for r in rows),
            }
    print(json.dumps({"records": records, "summary": summary}, indent=2))


if __name__ == "__main__":
    main()

benchmark_endpoints.py

#!/usr/bin/env python3
"""Matched streaming benchmark for local OpenAI-compatible endpoints."""

import json
import statistics
import time
import urllib.request


ENDPOINTS = {
    "1234": "http://127.0.0.1:1234/v1/chat/completions",
    "8081": "http://127.0.0.1:8081/llama/v1/chat/completions",
}

SCENARIOS = {
    "prefill": {
        "max_tokens": 32,
        "prompt": (
            "Analyze the following synthetic systems-design notes and then summarize the main tradeoffs. "
            + "Memory bandwidth controls weight streaming; quantization reduces bytes per parameter; "
              "batching increases arithmetic intensity; KV cache size grows with context; speculative "
              "decoding helps when draft acceptance is high; synchronization can limit small batches. " * 100
        ),
    },
    "generation": {
        "max_tokens": 256,
        "prompt": (
            "Write a detailed numbered explanation of GPU language-model inference. Continue until the "
            "token limit. Cover memory bandwidth, quantization, kernels, KV cache, batching, and speculative decoding."
        ),
    },
}


def percentile(values, p):
    ordered = sorted(values)
    index = (len(ordered) - 1) * p
    lower = int(index)
    upper = min(lower + 1, len(ordered) - 1)
    fraction = index - lower
    return ordered[lower] * (1 - fraction) + ordered[upper] * fraction


def run_stream(endpoint, scenario, trial):
    config = SCENARIOS[scenario]
    body = {
        "model": "qwen3.8-27b",
        "messages": [{
            "role": "user",
            # Put the nonce first: both servers use longest-common-prefix caches,
            # and a suffix nonce would allow nearly the whole prompt to be reused.
            "content": f"Benchmark nonce: {scenario}-{trial}-{endpoint}\n{config['prompt']}",
        }],
        "max_tokens": config["max_tokens"],
        "temperature": 0,
        "stream": True,
        "stream_options": {"include_usage": True},
        "cache_prompt": False,
    }
    request = urllib.request.Request(
        ENDPOINTS[endpoint],
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    started = time.perf_counter()
    first_token_at = None
    usage = None
    finish_reason = None
    chunks = 0
    with urllib.request.urlopen(request, timeout=180) as response:
        for raw_line in response:
            line = raw_line.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            event = json.loads(payload)
            if event.get("usage"):
                usage = event["usage"]
            for choice in event.get("choices", []):
                delta = choice.get("delta", {})
                produced = delta.get("content") or delta.get("reasoning_content")
                if produced:
                    chunks += 1
                    if first_token_at is None:
                        first_token_at = time.perf_counter()
                if choice.get("finish_reason"):
                    finish_reason = choice["finish_reason"]
    ended = time.perf_counter()
    if first_token_at is None or usage is None:
        raise RuntimeError(f"incomplete stream from {endpoint}: first={first_token_at}, usage={usage}")
    completion_tokens = usage["completion_tokens"]
    prompt_tokens = usage["prompt_tokens"]
    ttft = first_token_at - started
    generation_window = ended - first_token_at
    return {
        "endpoint": endpoint,
        "scenario": scenario,
        "trial": trial,
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "ttft_s": ttft,
        "total_s": ended - started,
        "generation_window_s": generation_window,
        "output_tokens_per_s": ((completion_tokens - 1) / generation_window) if generation_window > 0 else None,
        "end_to_end_tokens_per_s": completion_tokens / (ended - started),
        "finish_reason": finish_reason,
        "stream_chunks": chunks,
    }


def summarize(records):
    summary = {}
    for scenario in SCENARIOS:
        summary[scenario] = {}
        for endpoint in ENDPOINTS:
            rows = [r for r in records if r["scenario"] == scenario and r["endpoint"] == endpoint]
            metrics = {}
            for key in ("ttft_s", "total_s", "output_tokens_per_s", "end_to_end_tokens_per_s"):
                values = [r[key] for r in rows]
                metrics[key] = {
                    "mean": statistics.mean(values),
                    "median": statistics.median(values),
                    "stdev": statistics.stdev(values),
                    "min": min(values),
                    "max": max(values),
                    "p90": percentile(values, 0.90),
                }
            summary[scenario][endpoint] = {
                "n": len(rows),
                "prompt_tokens": sorted(set(r["prompt_tokens"] for r in rows)),
                "completion_tokens": sorted(set(r["completion_tokens"] for r in rows)),
                "metrics": metrics,
            }
    return summary


def main():
    records = []
    # Short warm-up for model/kernel readiness; excluded from reported results.
    for endpoint in ENDPOINTS:
        run_stream(endpoint, "generation", "warmup")

    # Alternate endpoint order on each trial to reduce order and thermal bias.
    for scenario in SCENARIOS:
        for trial in range(1, 6):
            order = ["1234", "8081"] if trial % 2 else ["8081", "1234"]
            for endpoint in order:
                result = run_stream(endpoint, scenario, trial)
                records.append(result)
                print(json.dumps({"progress": result}), flush=True)

    print(json.dumps({"records": records, "summary": summarize(records)}, indent=2))


if __name__ == "__main__":
    main()