Skip to content

Instantly share code, notes, and snippets.

@kevinsperrine

kevinsperrine/local-claude-code-omlx.md

Last active April 14, 2026 03:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save kevinsperrine/0ddcf80d30bf746dad4be60ec78aec9b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save kevinsperrine/0ddcf80d30bf746dad4be60ec78aec9b to your computer and use it in GitHub Desktop.
Download ZIP
Running Claude Code locally with Gemma 4 on Apple Silicon via oMLX
Raw
local-claude-code-omlx.md

Running Claude Code Locally with Gemma 4 on Apple Silicon

Run Claude Code entirely offline using Gemma 4 26B and oMLX on Apple Silicon Macs.

What This Gets You

  • Claude Code running against a local Gemma 4 model — no API keys, no cloud, no data leaving your machine
  • Native Anthropic API support — oMLX speaks Claude's protocol directly, no translation proxy needed
  • Tiered KV caching (RAM + SSD) for fast prompt processing across long sessions
  • Thinking/reasoning mode with proper streaming to Claude Code's UI
  • Tool calling (file edits, bash, grep, etc.) working out of the box
  • Auto-start on boot via macOS LaunchAgent

Architecture

Claude Code CLI
    │
    │ Anthropic /v1/messages API (port 8082)
    ▼
Sampling Proxy (~80 lines Python, recommended)
    │  - Temperature: 0.5 tools / 1.5 normal
    │  - Repetition penalty: 1.0 (disabled)
    │  - System prompt injection for local model tool guidance
    │  - Request/response logging
    │
    │ Anthropic /v1/messages API (port 8080)
    ▼
oMLX Server (native Anthropic + OpenAI API)
    │
    ▼
Gemma 4 26B-A4B MoE (8-bit, MLX on Apple Silicon GPU)

Why oMLX? It speaks the Anthropic API natively. Earlier approaches required an 800+ line proxy to translate between Anthropic and OpenAI API formats, handle tool call parsing edge cases, strip thinking token leaks, and manage streaming state. oMLX handles all of this internally with dedicated Gemma 4 support — tool call parsing, thinking token routing, context scaling, and tiered KV caching.

Why the sampling proxy? Local models like Gemma 4 need different sampling parameters than cloud Claude, and they benefit from system prompt injection that reminds them about tool usage patterns (read before edit, use Edit not Write, etc.). The proxy handles all of this transparently — no oMLX configuration changes needed.

Why Gemma 4 26B-A4B (MoE)?

  • Only 3.8B active parameters per token despite 26B total — fast inference
  • 8-bit quantization fits in 64 GB unified memory (~27 GB model)
  • Strong tool-calling with native call:name{args} format
  • Thinking mode via chat template with separated reasoning/content tokens
  • ~25-35 tok/s generation on Apple Silicon (varies by chip)

Important: Use 8-bit quantization. The 4-bit variant has known bugs with MLX's scaled_dot_product_attention that cause repetition loops under batch inference (ml-explore/mlx#3384). Stick with 8-bit unless you're very memory constrained.

Hardware Requirements

Component 64 GB (M1/M2/M3/M4) 128 GB (M4 Max/Ultra)
Model (8-bit) ~27 GB ~27 GB
Hot KV cache 8 GB 24 GB
SSD page cache 20+ GB disk 50+ GB disk
Context window 128k tokens 256k tokens
Max concurrent agents 2 8
Headroom for OS/apps ~25 GB ~70 GB

64 GB note: The 8-bit model fits but leaves less room for caching. You'll see ~20-30 tok/s on M1 Max, faster on newer chips. If memory is very tight, use a 4-bit variant (~14 GB) — see Model Variants.

Prerequisites

  • macOS 14.0+ (Sonoma or later)
  • Homebrew package manager
  • Node.js 18+ (for Claude Code CLI)
  • ~15 GB free disk for the model download
  • ~20 GB free disk for SSD KV cache

Installation

Step 1: Install Claude Code

npm install -g @anthropic-ai/claude-code

Step 2: Install oMLX

Option A: Homebrew (recommended — macOS Tahoe / 26.0+)

brew install jundot/omlx/omlx

That's it. Homebrew handles Python 3.11, the virtual environment, and all dependencies.

Note: The brew formula compiles the tokenizers Rust crate from source, which requires Xcode CLI Tools 26.3+. This ships with macOS Tahoe. On older macOS versions (Sequoia and earlier), the build will fail with a linker error. Use Option B instead.

Option B: pip (macOS Sonoma / Sequoia / pre-Tahoe)

If you're on an older macOS or the brew install fails, install from source into a virtual environment:

# Create a dedicated venv (Python 3.11 recommended)
python3.11 -m venv ~/.omlx-venv
source ~/.omlx-venv/bin/activate

# Install oMLX with MCP support
pip install "omlx[mcp] @ git+https://github.com/jundot/omlx.git"

deactivate

Troubleshooting: If pip install fails building the tokenizers wheel, install it separately first with pip install tokenizers (uses a pre-built binary wheel), then retry the oMLX install.

Step 3: Download a model and set up the model directory

# Download Gemma 4 26B 8-bit (~14 GB download, ~27 GB on disk)
omlx pull mlx-community/gemma-4-26b-a4b-it-8bit

# Create the model directory and symlink
mkdir -p ~/.omlx/models ~/.omlx/cache
ln -s $(omlx model-path mlx-community/gemma-4-26b-a4b-it-8bit) \
    ~/.omlx/models/gemma-4-26b-8bit

If omlx pull isn't available (or you used the pip install), download manually:

pip install huggingface-hub  # if not already installed
huggingface-cli download mlx-community/gemma-4-26b-a4b-it-8bit
ln -s $(python3 -c "from huggingface_hub import snapshot_download; print(snapshot_download('mlx-community/gemma-4-26b-a4b-it-8bit', local_files_only=True))") \
    ~/.omlx/models/gemma-4-26b-8bit

Verify the model is linked:

ls ~/.omlx/models/gemma-4-26b-8bit/
# Should list model files: config.json, model*.safetensors, tokenizer.json, etc.

Step 5: Configure oMLX

Create ~/.omlx/settings.json:

{
  "version": "1.0",
  "server": {
    "host": "127.0.0.1",
    "port": 8080,
    "log_level": "info",
    "cors_origins": ["*"]
  },
  "model": {
    "model_dirs": ["~/.omlx/models"],
    "model_dir": "~/.omlx/models",
    "max_model_memory": "auto",
    "model_fallback": false
  },
  "memory": {
    "max_process_memory": "auto",
    "prefill_memory_guard": true
  },
  "scheduler": {
    "max_concurrent_requests": 2
  },
  "cache": {
    "enabled": true,
    "ssd_cache_dir": "~/.omlx/cache",
    "ssd_cache_max_size": "auto",
    "hot_cache_max_size": "8GB",
    "initial_cache_blocks": 128
  },
  "sampling": {
    "max_context_window": 131072,
    "max_tokens": 8192,
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 64,
    "repetition_penalty": 1.0
  },
  "claude_code": {
    "context_scaling_enabled": true,
    "target_context_size": 120000,
    "mode": "local",
    "opus_model": "gemma-4-26b-8bit",
    "sonnet_model": "gemma-4-26b-8bit",
    "haiku_model": "gemma-4-26b-8bit"
  },
  "auth": {
    "api_key": null,
    "skip_api_key_verification": false
  },
  "ui": {
    "language": "en"
  }
}

Note: If oMLX doesn't expand ~, replace ~/.omlx with the full path /Users/YOUR_USERNAME/.omlx throughout.

Settings for 128 GB machines

If you have 128 GB unified memory, change these values for better performance:

Setting 64 GB 128 GB
scheduler.max_concurrent_requests 2 8
cache.hot_cache_max_size "8GB" "24GB"
cache.initial_cache_blocks 128 256
sampling.max_context_window 131072 262144
claude_code.target_context_size 120000 200000

Step 6: Start oMLX as a service

If you installed via Homebrew (Option A):

# Start oMLX now and auto-start on login
brew services start omlx

Other service commands:

brew services stop omlx      # Stop the server
brew services restart omlx    # Restart (picks up settings.json changes)
brew services info omlx       # Check status

If you installed via pip (Option B):

Create a LaunchAgent so oMLX starts automatically on login. Create ~/Library/LaunchAgents/com.omlx.server.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.omlx.server</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/YOUR_USERNAME/.omlx-venv/bin/omlx</string>
        <string>serve</string>
        <string>--model-dir</string>
        <string>/Users/YOUR_USERNAME/.omlx/models</string>
        <string>--port</string>
        <string>8080</string>
        <string>--host</string>
        <string>127.0.0.1</string>
        <string>--paged-ssd-cache-dir</string>
        <string>/Users/YOUR_USERNAME/.omlx/cache</string>
        <string>--hot-cache-max-size</string>
        <string>8GB</string>
        <string>--log-level</string>
        <string>info</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/omlx-server.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/omlx-server.log</string>
    <key>ProcessType</key>
    <string>Background</string>
</dict>
</plist>

Replace YOUR_USERNAME with your macOS username. For 128 GB machines, change 8GB to 24GB.

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.omlx.server.plist

Verify it's running (either method):

curl -s http://127.0.0.1:8080/v1/models
# Expected: {"object":"list","data":[{"id":"gemma-4-26b-8bit","object":"model",...}]}

First launch: The model loads into memory on the first request (~30-60 seconds). Subsequent requests benefit from caching.

Step 7: Create the shell function

Add to your ~/.zshrc or ~/.bashrc:

# Run Claude Code against a local Gemma 4 model via oMLX
# Usage: claude-local [any claude args...]
claude-local() {
    local model="gemma-4-26b-8bit"

    # Check that oMLX is running
    if ! curl -sf http://127.0.0.1:8080/v1/models > /dev/null 2>&1; then
        echo "oMLX not running."
        echo "  Check: brew services info omlx"
        echo "  Logs:  tail -20 /tmp/omlx-server.log"
        echo "  Start: brew services start omlx"
        return 1
    fi

    # Clear any existing Anthropic credentials so they don't interfere
    unset ANTHROPIC_AUTH_TOKEN
    unset ANTHROPIC_API_KEY

    # Start sampling proxy if not running (adjusts temp, injects system prompt)
    if ! curl -sf http://127.0.0.1:8082/v1/models > /dev/null 2>&1; then
        echo "Starting sampling proxy (8082 → 8080)..."
        nohup python3 ~/.config/omlx-temp-proxy/proxy.py --port 8082 --backend-port 8080 \
            > /tmp/omlx-temp-proxy.log 2>&1 &
        sleep 1
    fi

    # Point Claude Code at sampling proxy → oMLX
    ANTHROPIC_BASE_URL="http://127.0.0.1:8082" \
    ANTHROPIC_AUTH_TOKEN="none" \
    ANTHROPIC_MODEL="$model" \
    ANTHROPIC_CUSTOM_MODEL_OPTION="$model" \
    ANTHROPIC_CUSTOM_MODEL_OPTION_NAME="Gemma 4 26B (oMLX Local)" \
    ANTHROPIC_DEFAULT_OPUS_MODEL="$model" \
    ANTHROPIC_DEFAULT_OPUS_MODEL_SUPPORTED_CAPABILITIES="thinking,interleaved_thinking" \
    ANTHROPIC_DEFAULT_SONNET_MODEL="$model" \
    ANTHROPIC_DEFAULT_SONNET_MODEL_SUPPORTED_CAPABILITIES="thinking,interleaved_thinking" \
    ANTHROPIC_DEFAULT_HAIKU_MODEL="$model" \
    ANTHROPIC_DEFAULT_HAIKU_MODEL_SUPPORTED_CAPABILITIES="thinking,interleaved_thinking" \
    CLAUDE_CODE_SUBAGENT_MODEL="$model" \
    CLAUDE_CODE_AUTO_COMPACT_WINDOW="131072" \
    CLAUDE_AUTOCOMPACT_PCT_OVERRIDE="75" \
    CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY="2" \
    CLAUDE_CODE_MAX_OUTPUT_TOKENS="8192" \
    CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS="8000" \
    CLAUDE_CODE_MAX_RETRIES="3" \
    API_TIMEOUT_MS="30000000" \
    BASH_DEFAULT_TIMEOUT_MS="2400000" \
    BASH_MAX_TIMEOUT_MS="2500000" \
    TASK_MAX_OUTPUT_LENGTH="32768" \
    CLAUDE_CODE_DISABLE_1M_CONTEXT="1" \
    CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING="1" \
    CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1" \
    CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS="1" \
    CLAUDE_CODE_ATTRIBUTION_HEADER="0" \
    DISABLE_PROMPT_CACHING="1" \
    DISABLE_COST_WARNINGS="1" \
    DISABLE_TELEMETRY="1" \
    DISABLE_ERROR_REPORTING="1" \
    DISABLE_EXTRA_USAGE_COMMAND="1" \
    DISABLE_LOGIN_COMMAND="1" \
    DISABLE_LOGOUT_COMMAND="1" \
    CLAUDE_ENABLE_STREAM_WATCHDOG="1" \
    CLAUDE_STREAM_IDLE_TIMEOUT_MS="600000" \
    MAX_STRUCTURED_OUTPUT_RETRIES="10" \
    claude "$@"
}

128 GB adjustments

For 128 GB machines, change these values in the function above:

CLAUDE_CODE_AUTO_COMPACT_WINDOW="262144"
CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY="8"

Step 8: Run it

source ~/.zshrc
claude-local

You should see Claude Code start with "Gemma 4 26B (oMLX Local)" as the model name.

Step 9: Verify the full stack

# Test a direct API request
curl -s http://127.0.0.1:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: none" \
  -d '{
    "model": "gemma-4-26b-8bit",
    "max_tokens": 50,
    "stream": false,
    "messages": [{"role": "user", "content": "Hello! Say hi in one sentence."}]
  }' | python3 -m json.tool

You should see a JSON response with the model's reply in content[0].text.

Environment Variables Reference

These are all the env vars set in claude-local() and what each does:

Core Routing

Variable Value Purpose
ANTHROPIC_BASE_URL http://127.0.0.1:8080 Redirects all API calls to local oMLX
ANTHROPIC_AUTH_TOKEN none Satisfies auth check without real credentials
ANTHROPIC_MODEL Model ID Primary model for Claude Code
ANTHROPIC_CUSTOM_MODEL_OPTION Model ID Model identifier for Claude Code UI
ANTHROPIC_CUSTOM_MODEL_OPTION_NAME Display name Human-readable name shown in Claude Code
ANTHROPIC_DEFAULT_*_MODEL Model ID Routes all model tiers (opus/sonnet/haiku) to local
CLAUDE_CODE_SUBAGENT_MODEL Model ID Ensures subagents also use local model

Thinking Support

Variable Value Purpose
ANTHROPIC_DEFAULT_OPUS_MODEL_SUPPORTED_CAPABILITIES thinking,interleaved_thinking Tells Claude Code opus tier supports thinking
ANTHROPIC_DEFAULT_SONNET_MODEL_SUPPORTED_CAPABILITIES thinking,interleaved_thinking Same for sonnet tier
ANTHROPIC_DEFAULT_HAIKU_MODEL_SUPPORTED_CAPABILITIES thinking,interleaved_thinking Same for haiku tier

These make Claude Code send thinking: { type: 'enabled', budget_tokens: N } in requests. oMLX handles the thinking protocol natively with Gemma 4's chat template. Thinking output appears in Claude Code's UI (toggle with Ctrl+O).

Context & Output Limits

Variable Value Purpose
CLAUDE_CODE_AUTO_COMPACT_WINDOW 131072 / 262144 Context window size for auto-compaction trigger
CLAUDE_AUTOCOMPACT_PCT_OVERRIDE 75 Compact at 75% context usage
CLAUDE_CODE_MAX_OUTPUT_TOKENS 8192 Max tokens per response
CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS 8000 Max tokens for file reads
TASK_MAX_OUTPUT_LENGTH 32768 Max output for task/subagent results
MAX_STRUCTURED_OUTPUT_RETRIES 10 Retries for structured output parsing failures

Timeouts

Variable Value Purpose
API_TIMEOUT_MS 30000000 (~8 hrs) Prevents API timeout on long local generations
BASH_DEFAULT_TIMEOUT_MS 2400000 (40 min) Default timeout for shell commands
BASH_MAX_TIMEOUT_MS 2500000 (42 min) Max timeout for shell commands
CLAUDE_STREAM_IDLE_TIMEOUT_MS 600000 (10 min) Stream watchdog timeout — Gemma's thinking phase can be long and silent
CLAUDE_ENABLE_STREAM_WATCHDOG 1 Enables stream watchdog (kills dead connections)

Parallel Agents

Variable Value Purpose
CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY 2 / 8 Max parallel tool calls / agent requests

Disabled Features

These features require a connection to Anthropic's servers and don't work with local inference:

Variable Value Purpose
CLAUDE_CODE_DISABLE_1M_CONTEXT 1 Disables 1M context (Anthropic-only)
CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING 1 Disables adaptive thinking (Anthropic-only)
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC 1 No background pings to Anthropic
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS 1 Disables beta features requiring Anthropic
CLAUDE_CODE_ATTRIBUTION_HEADER 0 Disables attribution header
DISABLE_PROMPT_CACHING 1 Disables Anthropic's server-side prompt caching (oMLX has its own)
DISABLE_COST_WARNINGS 1 No cost warnings (it's free locally)
DISABLE_TELEMETRY 1 No telemetry sent anywhere
DISABLE_ERROR_REPORTING 1 No error reporting to Anthropic
DISABLE_LOGIN_COMMAND 1 Hides login command (no auth needed)
DISABLE_LOGOUT_COMMAND 1 Hides logout command

oMLX Settings Reference

Sampling Parameters

Parameter Value Why
temperature 1.0 Google's official recommendation. The proxy overrides to 1.5 normal / 0.5 tools.
top_p 0.95 Nucleus sampling per Gemma 4 model card
top_k 64 Top-k sampling per Gemma 4 model card
repetition_penalty 1.0 Disabled. Token-level penalty is ineffective against paragraph-level loops (google-deepmind/gemma#622). The proxy enforces 1.0.
max_tokens 8192 Max generation length per request. Caps worst-case loops.
max_context_window 131072 / 262144 Model's context capacity. Gemma 4 supports up to 256k.

Cache Settings

Parameter Value Why
hot_cache_max_size 8GB / 24GB In-memory KV cache. More = faster prompt processing.
ssd_cache_dir ~/.omlx/cache Where overflow KV cache is paged to disk
ssd_cache_max_size auto oMLX manages SSD cache size automatically
initial_cache_blocks 128 / 256 Pre-allocated cache blocks. More = fewer reallocations.

Context Scaling

Parameter Value Why
context_scaling_enabled true Reports scaled token counts to Claude Code so auto-compact triggers at the right time relative to the real context limit
target_context_size 120000 / 200000 What oMLX reports as the context size to Claude Code

Recommended: Sampling Proxy

A lightweight proxy (~80 lines) that sits between Claude Code and oMLX. Strongly recommended — it fixes several issues with running local models:

  1. Dynamic temperature: 0.5 for tool calls (tight structured JSON), 1.5 for normal text (higher entropy prevents Gemma 4 repetition loops)
  2. Repetition penalty override: Forces repetition_penalty: 1.0 (disabled) on every request — token-level penalties are proven ineffective against Gemma 4's paragraph-level loops
  3. System prompt injection: Appends tool usage rules (read before edit, use Edit not Write, parameter names) that local models need but cloud Claude handles natively
  4. Request/response logging: Logs to /tmp/omlx-proxy.log for debugging tool calls, streaming health, and agent team coordination

Setup

# Install dependencies
pip install fastapi uvicorn httpx

# Create proxy directory
mkdir -p ~/.config/omlx-temp-proxy

Create ~/.config/omlx-temp-proxy/proxy.py:

"""
Lightweight sampling proxy for oMLX.

Request-body overrides (applied to /v1/messages POST):
  - temperature: 0.5 with tools, 1.5 without
  - repetition_penalty: 1.0 (disabled — ineffective per gemma#622)
  - system prompt: appends local-model guidance (read before edit, etc.)

Logging:
  - Requests and responses logged to /tmp/omlx-proxy.log
  - tail -f /tmp/omlx-proxy.log to watch in real time

Usage:
  python proxy.py                          # default: listen 8082 → oMLX 8080
  python proxy.py --port 8082 --backend-port 8080
"""

import argparse
import json
import logging
import time
from logging.handlers import RotatingFileHandler

import httpx
import uvicorn
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, Response

LOG_FILE = "/tmp/omlx-proxy.log"

log = logging.getLogger("omlx-temp-proxy")
log.setLevel(logging.DEBUG)
_fh = RotatingFileHandler(LOG_FILE, maxBytes=10 * 1024 * 1024, backupCount=3)
_fh.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
log.addHandler(_fh)

app = FastAPI()

BACKEND_URL = "http://127.0.0.1:8080"

TEMP_WITH_TOOLS = 0.5
TEMP_WITHOUT_TOOLS = 1.5

# Extra system instructions appended to every request.
# Compensates for local model quirks that cloud models handle natively.
SYSTEM_SUFFIX = """
## MANDATORY Tool Rules — violations cause failures

### Read Before Edit — THIS IS THE MOST IMPORTANT RULE

EVERY call to Edit REQUIRES a prior call to Read on the SAME file in the SAME conversation.
The Edit tool checks whether you have read the file. If you have not, it WILL reject your edit.

CORRECT workflow:
1. Call Read with the file_path
2. Find the exact text you want to change in the Read output
3. Copy that exact text into old_string (character-for-character, including whitespace)
4. Call Edit with file_path, old_string, new_string

WRONG (will fail):
- Calling Edit without calling Read first
- Guessing what old_string should be without reading the file
- Reading file A then editing file B (you must Read each file you Edit)
- Using Edit with old_string that doesn't match the file exactly

If you are about to call Edit, stop and ask yourself: "Did I Read this file?" If no, Read it first.

### Small Edits, Not Whole Files
- Use Edit for changes to existing files. Do NOT use Write to replace entire files when only a few lines change.
- Only use Write for NEW files that don't exist yet, or when truly every line must change.

### Always Use Tools
- NEVER output code as text instead of using tools to write/edit files.
- NEVER suggest the user run commands — execute them yourself with Bash.
- Take action immediately.

### Tool Parameter Names (snake_case, not camelCase)
- Edit: `file_path`, `old_string`, `new_string`, `replace_all`
- Read: `file_path`, `offset`, `limit`
- Write: `file_path`, `content`
- Grep: `pattern`, `path`, `glob`, `output_mode`
- Glob: `pattern`, `path`
- Bash: `command`, `description` (both required)

### Avoid These Mistakes
- Do NOT call tools that do not exist. Use Bash as a fallback.
- Do NOT pass arrays or objects as strings — use proper JSON types.
- Make independent tool calls in parallel, not sequentially.
""".strip()


def _summarize_messages(messages: list) -> str:
    """One-line-per-message summary for the log."""
    lines = []
    for msg in messages[-5:]:
        role = msg.get("role", "?")
        content = msg.get("content", "")
        if isinstance(content, str):
            preview = content[:120].replace("\n", " ")
        elif isinstance(content, list):
            parts = []
            for block in content:
                if isinstance(block, dict):
                    if block.get("type") == "text":
                        parts.append(block.get("text", "")[:80])
                    elif block.get("type") == "tool_use":
                        parts.append(f"[tool_use:{block.get('name','')}]")
                    elif block.get("type") == "tool_result":
                        parts.append(f"[tool_result:{block.get('tool_use_id','')[:12]}]")
                    else:
                        parts.append(f"[{block.get('type', '?')}]")
            preview = " | ".join(parts)[:120]
        else:
            preview = str(content)[:120]
        lines.append(f"    {role}: {preview}")
    total = len(messages)
    if total > 5:
        lines.insert(0, f"    ... ({total - 5} earlier messages omitted)")
    return "\n".join(lines)


def _summarize_tools(tools: list) -> str:
    """List tool names provided in the request."""
    return ", ".join(t.get("name", "?") for t in tools)


@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE"])
async def proxy(request: Request, path: str):
    body = await request.body()
    is_streaming = False
    t_start = time.perf_counter()

    if request.method == "POST" and "messages" in path:
        data = json.loads(body)
        has_tools = bool(data.get("tools"))
        data["temperature"] = TEMP_WITH_TOOLS if has_tools else TEMP_WITHOUT_TOOLS
        data["repetition_penalty"] = 1.0

        # Append local-model guidance to system prompt
        system = data.get("system", "")
        if isinstance(system, str):
            data["system"] = f"{system}\n\n{SYSTEM_SUFFIX}" if system else SYSTEM_SUFFIX
        elif isinstance(system, list):
            data["system"].append({"type": "text", "text": SYSTEM_SUFFIX})

        is_streaming = data.get("stream", False)

        tools_summary = _summarize_tools(data["tools"]) if has_tools else "(none)"
        log.info("=" * 60)
        log.info("REQUEST  %s /%s", request.method, path)
        log.info("  model:   %s", data.get("model", "?"))
        log.info("  temp:    %s  rep_penalty: 1.0  stream: %s", data["temperature"], is_streaming)
        log.info("  tools:   %s", tools_summary)
        log.info("  max_tok: %s", data.get("max_tokens", "?"))
        log.info("  messages:\n%s", _summarize_messages(data.get("messages", [])))

        body = json.dumps(data).encode()
    else:
        log.debug("PASSTHROUGH %s /%s", request.method, path)

    headers = {
        k: v for k, v in request.headers.items()
        if k.lower() not in ("host", "content-length")
    }

    url = f"{BACKEND_URL}/{path}"

    if is_streaming:
        # IMPORTANT: Client lifecycle is managed by the generator, NOT the handler.
        # Using `async with` here would close the client when the handler returns,
        # before FastAPI starts iterating the StreamingResponse generator — resulting
        # in an empty stream. Instead, we create the client here and close it in the
        # generator's `finally` block.
        client = httpx.AsyncClient(timeout=httpx.Timeout(300.0))
        req = client.build_request(
            request.method, url, content=body, headers=headers
        )
        resp = await client.send(req, stream=True)

        async def stream():
            chunks_seen = 0
            text_accum = []
            tool_calls = []
            try:
                async for chunk in resp.aiter_bytes():
                    chunks_seen += 1
                    try:
                        for line in chunk.decode("utf-8").split("\n"):
                            if not line.startswith("data: "):
                                continue
                            evt = json.loads(line[6:])
                            evt_type = evt.get("type", "")
                            if evt_type == "content_block_delta":
                                delta = evt.get("delta", {})
                                if delta.get("type") == "text_delta":
                                    text_accum.append(delta.get("text", ""))
                            elif evt_type == "content_block_start":
                                block = evt.get("content_block", {})
                                if block.get("type") == "tool_use":
                                    tool_calls.append(block.get("name", "?"))
                    except (json.JSONDecodeError, UnicodeDecodeError):
                        pass
                    yield chunk
            except httpx.ReadError:
                pass
            finally:
                await resp.aclose()
                await client.aclose()
                elapsed = time.perf_counter() - t_start
                output_text = "".join(text_accum)
                preview = output_text[:300].replace("\n", " ")
                log.info("RESPONSE stream  %d chunks  %.1fs", chunks_seen, elapsed)
                if tool_calls:
                    log.info("  tool_use: %s", ", ".join(tool_calls))
                if preview:
                    log.info("  text: %s%s", preview, "..." if len(output_text) > 300 else "")

        return StreamingResponse(
            stream(),
            status_code=resp.status_code,
            headers=dict(resp.headers),
        )
    else:
        async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
            resp = await client.request(
                request.method, url, content=body, headers=headers
            )
            elapsed = time.perf_counter() - t_start
            log.info("RESPONSE %d  %.1fs  %d bytes", resp.status_code, elapsed, len(resp.content))
            return Response(
                content=resp.content,
                status_code=resp.status_code,
                headers=dict(resp.headers),
            )


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--port", type=int, default=8082)
    parser.add_argument("--backend-port", type=int, default=8080)
    args = parser.parse_args()
    BACKEND_URL = f"http://127.0.0.1:{args.backend_port}"
    uvicorn.run(app, host="127.0.0.1", port=args.port)

Critical streaming note: The streaming path creates httpx.AsyncClient outside of async with and closes it in the generator's finally block. This is intentional — if you use async with, the client closes when the handler returns (before FastAPI starts iterating the generator), resulting in an empty stream. This is a common FastAPI streaming proxy pitfall.

Using the proxy

# Start the proxy (runs in background)
nohup python3 ~/.config/omlx-temp-proxy/proxy.py --port 8082 --backend-port 8080 \
    > /tmp/omlx-temp-proxy.log 2>&1 &

# Watch logs in real time
tail -f /tmp/omlx-proxy.log

Auto-starting the proxy

Add this block to your claude-local function before the ANTHROPIC_BASE_URL line:

    # Start sampling proxy if not running
    if ! curl -sf http://127.0.0.1:8082/v1/models > /dev/null 2>&1; then
        echo "Starting sampling proxy (8082 → 8080)..."
        nohup python3 ~/.config/omlx-temp-proxy/proxy.py --port 8082 --backend-port 8080 \
            > /tmp/omlx-temp-proxy.log 2>&1 &
        sleep 1
    fi

Then set ANTHROPIC_BASE_URL="http://127.0.0.1:8082" in the function.

Customizing the system prompt

Edit the SYSTEM_SUFFIX variable in proxy.py to add or modify rules. Common additions:

  • Project-specific conventions ("always use Effect-TS patterns")
  • Additional tool guidance based on observed failures
  • Output style preferences

Changes take effect after restarting the proxy (pkill -f omlx-temp-proxy; nohup python3 ...).

Model Variants

All models are from the mlx-community HuggingFace organization (pre-quantized for MLX):

Model Disk/Memory Active Params Context Best for
gemma-4-26b-a4b-it-8bit ~27 GB 3.8B (MoE) 256k Best quality. Recommended for 64-128 GB.
gemma-4-26b-a4b-it-4bit ~14 GB 3.8B (MoE) 256k Tight memory. Good quality, smaller footprint.
gemma-4-4b-it-8bit ~8 GB 4B (Dense) 128k Fastest. Good for simple tasks or very constrained memory.

To switch models:

  1. Download: huggingface-cli download mlx-community/MODEL_NAME
  2. Symlink into ~/.omlx/models/ with a short name
  3. Update model names in ~/.omlx/settings.json under claude_code
  4. Update the model variable in your claude-local function
  5. Restart oMLX: brew services restart omlx

Tuning Guide

Repetition Loops

Gemma 4 has a known model-level tendency toward repetition that is a regression from Gemma 3. This is the most common issue you'll encounter. The model generates a paragraph of reasoning, then generates the same paragraph again, over and over.

Root causes (layered):

  1. Model-level: Gemma 4 has inherent token-doubling behavior. Tested with identical seeds, Gemma 4 MoE fails 5/10 while Gemma 3 shows 0/10 failures (google-deepmind/gemma#622).
  2. Quantization amplifier: 4-bit quantization makes it dramatically worse via SDPA numerical drift in MLX (ml-explore/mlx#3384). Use 8-bit.
  3. Low temperature: Lower temperature pushes the model toward deterministic token choices, making loops more likely. Google recommends temp 1.0; the community reports 1.5 works better for coding.
  4. Grammar constraints: JSON schema output triggers loops more aggressively (10/10 failure rate in testing for long JSON).

What does NOT work:

  • repetition_penalty — tested at 1.0, 1.15, and 1.5 with identical results on identical seeds. Token-level penalties cannot catch paragraph-level loops because the repeating unit contains enough token diversity. Google doesn't recommend it. Keep at 1.0.
  • DRY sampling — catches sequence-level repetition but is only available in llama.cpp, not MLX/oMLX.

What helps (in order of impact):

  1. Use 8-bit, not 4-bit — avoids the MLX SDPA mask divergence bug that traps the sampler in cyclic repetition.
  2. Higher temperature (1.5) — gives the model more entropy to escape repetition attractors. The sampling proxy handles this automatically.
  3. max_tokens: 8192 — caps worst-case loop to ~4 minutes at 30 tok/s. Lower to 4096 if loops are frequent.
  4. Sampling proxy with system prompt injection — the proxy's tool usage rules keep the model focused on concrete actions rather than open-ended reasoning where loops are most common.

Relevant issues:

Memory Pressure (64 GB)

If you see slowdowns, system swap usage, or memory_pressure warnings:

  1. Reduce hot cache: "hot_cache_max_size": "4GB" — SSD cache compensates, just slower
  2. Reduce context window: "max_context_window": 65536 (64k) — fewer KV cache entries
  3. Lower concurrent requests: "max_concurrent_requests": 1 — less parallel memory usage
  4. Use 4-bit model: gemma-4-26b-a4b-it-4bit saves ~13 GB of memory
  5. Close memory-hungry apps — browsers (especially Chrome), Docker, Xcode, Simulator

Performance Tips

  • SSD cache is your friend — oMLX pages KV cache to SSD when RAM is tight. Make sure you have 20+ GB free disk space. NVMe SSDs on Apple Silicon are fast enough that SSD-cached prompts still process quickly.
  • First request is slow — the model loads into GPU memory on first use (~30-60s). Subsequent requests are fast.
  • Context scaling — oMLX's context_scaling_enabled reports scaled token counts so Claude Code's auto-compact triggers at the right time. Don't disable this.
  • Prompt caching — oMLX caches prompt prefixes. In agent workflows with repeated system prompts, cache hit rates can exceed 90%, dramatically speeding up prefill.

Performance Benchmarks

Measured with Gemma 4 26B-A4B 8-bit on M4 Max 128 GB, 24 GB hot cache:

Metric 1 Request 2 Parallel 4 Parallel
Generation speed ~35 tok/s ~33 tok/s each ~25 tok/s each
Total throughput ~35 tok/s ~66 tok/s (1.88x) ~100 tok/s (~2.8x)
Model RSS ~27 GB ~27 GB ~27 GB

Key observations:

  • Memory does not increase with concurrency. Model weights (~27 GB) are shared. KV caches for concurrent requests add minimal overhead.
  • Throughput scales sublinearly — diminishing returns because memory bandwidth (not compute) is the bottleneck on Apple Silicon.
  • 64 GB machines will see ~20-30 tok/s for single requests. Parallel throughput is limited by the lower hot cache and concurrent request cap.

Known Issues & Limitations

Certificate Errors on Corporate Networks (Zscaler / SSL Inspection)

If you see certificate verify failed when downloading models, your corporate proxy (Zscaler, Netskope, etc.) is doing SSL inspection with its own root CA that Python doesn't trust.

Step 1: Export the corporate root CA from macOS Keychain

# Find the certificate name (usually "Zscaler Root CA" or similar)
security find-certificate -a -c "Zscaler" /Library/Keychains/System.keychain

# Export it as PEM
security find-certificate -a -c "Zscaler" -p /Library/Keychains/System.keychain \
    > ~/zscaler-root-ca.pem

If you don't know the certificate name, open Keychain Access, select System keychain, filter by "Certificates", and look for your proxy vendor's root CA. Right-click → Export as .pem.

Step 2: Create a combined CA bundle

# Find Python's default CA bundle
python3 -c "import certifi; print(certifi.where())"
# e.g., /opt/homebrew/lib/python3.11/site-packages/certifi/cacert.pem

# Create a combined bundle (system CAs + corporate CA)
cat "$(python3 -c 'import certifi; print(certifi.where())')" ~/zscaler-root-ca.pem \
    > ~/.ssl/corporate-ca-bundle.pem

mkdir -p ~/.ssl

Step 3: Set environment variables

Add these to your ~/.zshrc or ~/.bashrc:

export REQUESTS_CA_BUNDLE="$HOME/.ssl/corporate-ca-bundle.pem"
export SSL_CERT_FILE="$HOME/.ssl/corporate-ca-bundle.pem"
export CURL_CA_BUNDLE="$HOME/.ssl/corporate-ca-bundle.pem"
export NODE_EXTRA_CA_CERTS="$HOME/.ssl/corporate-ca-bundle.pem"

REQUESTS_CA_BUNDLE covers Python's requests library (used by huggingface-hub). SSL_CERT_FILE covers Python's httpx and ssl module (used by hf-xet downloads). CURL_CA_BUNDLE covers curl and tools that use libcurl. NODE_EXTRA_CA_CERTS covers Node.js (used by Claude Code itself).

After sourcing your shell config, huggingface-cli download and omlx pull should work behind the corporate proxy.

Web Search Returns No Results

Claude Code's web search uses Anthropic's server-side search API, which requires a valid API key. With ANTHROPIC_AUTH_TOKEN="none", searches fail silently. No workaround — this is a fundamental limitation of local inference.

Long "Thinking" Pauses

Gemma 4 can generate thousands of thinking tokens before producing visible content. At ~30 tok/s, a 4,000-token thinking chain takes ~2 minutes of apparent silence in the UI. The stream watchdog timeout is set to 10 minutes (CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000) to accommodate this. If you still see timeouts, increase the value.

Thinking Tokens Leaking into Output

If you see "Wait, I should..." or "Actually, let me..." deliberation text in the regular (non-thinking) output, Gemma is emitting thinking tokens in the wrong channel. This can happen with older oMLX versions. Update oMLX:

# Homebrew install:
brew upgrade omlx
brew services restart omlx

# pip install:
source ~/.omlx-venv/bin/activate
pip install --upgrade "omlx[mcp] @ git+https://github.com/jundot/omlx.git"
deactivate
launchctl kickstart -k gui/$(id -u)/com.omlx.server

RotatingKVCache Warnings

RotatingKVCache offset 44032 may be stale (buffer seq_len=1024, max_size=1024)

These appear at high context lengths when the KV cache's rotating buffer hits its boundary. They're warnings, not errors, but excessive spam can indicate the model is at the edge of its cache capacity. Reduce max_context_window or increase hot_cache_max_size if you have memory headroom.

Address Already in Use (Port 8080)

ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8080)

Something else is using port 8080. Find and stop it:

lsof -i :8080
# Then either stop that process or change oMLX's port in settings.json and the plist

Agent Teams Mailbox Bug

Claude Code's experimental agent teams feature has a known bug where the team-lead's mailbox polling reads inboxes/team-lead.json but teammates write to inboxes/team-lead-TEAMNAME.json. This is a Claude Code bug, not a local inference issue — it happens with the Anthropic API too.

Operations

Starting & Stopping

Homebrew install:

brew services start omlx       # Start and enable auto-start
brew services stop omlx        # Stop the server
brew services restart omlx     # Restart (picks up settings.json changes)
brew services info omlx        # Check status

pip install:

# Start (via LaunchAgent)
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.omlx.server.plist

# Restart
launchctl kickstart -k gui/$(id -u)/com.omlx.server

# Stop
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.omlx.server.plist

Either method:

# Stop the temperature proxy (if using)
pkill -f "omlx-temp-proxy/proxy.py"

Monitoring

# Check if oMLX is running and which model is loaded
curl -s http://127.0.0.1:8080/v1/models | python3 -m json.tool

# Memory and CPU usage
ps aux | grep omlx | grep -v grep | awk '{printf "PID: %s | CPU: %s%% | MEM: %s%% | RSS: %dMB\n", $2, $3, $4, $6/1024}'

# Watch server logs live
tail -f /tmp/omlx-server.log

# Watch proxy logs live (request/response details, tool calls, timing)
tail -f /tmp/omlx-proxy.log

# Check active connections to oMLX
lsof -i :8080 -P | grep ESTABLISHED | wc -l

Healthy Log Output

When things are working, oMLX logs look like:

omlx.server - INFO - Anthropic message: 512 tokens in 6.84s (74.8 tok/s)
omlx.server - INFO - Anthropic message: 256 tokens in 3.96s (64.7 tok/s)
omlx.scheduler - INFO - Using boundary cache snapshot for ...: storing 60416/60447 tokens

Logs

Log Location
oMLX server /tmp/omlx-server.log
Temperature proxy /tmp/omlx-temp-proxy.log

Updating oMLX

# Homebrew:
brew upgrade omlx
brew services restart omlx

# pip:
source ~/.omlx-venv/bin/activate
pip install --upgrade "omlx[mcp] @ git+https://github.com/jundot/omlx.git"
deactivate
launchctl kickstart -k gui/$(id -u)/com.omlx.server

Quick Reference

# Start Claude Code locally
claude-local

# Check oMLX status
curl -s http://127.0.0.1:8080/v1/models

# Restart oMLX (brew)
brew services restart omlx

# Restart oMLX (pip / LaunchAgent)
launchctl kickstart -k gui/$(id -u)/com.omlx.server

# View logs
tail -f /tmp/omlx-server.log

# Test a direct API call
curl -s http://127.0.0.1:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: none" \
  -d '{"model":"gemma-4-26b-8bit","max_tokens":50,"stream":false,"messages":[{"role":"user","content":"Hello!"}]}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment