server.md

Build chat completions plugin

Bulid this plugin. It should add a llm chat-completions-server command which starts a web server on port 8002 (or -p/--port X) which provides a clone of the classic OpenAI Chat Completions API endpoint that uses LLM async from the current in-development version of the ~/dev/llm library - run git diff against main in that folder to see what has changed there. Any questions?

24 previous messages

No questions yet—I’ll inspect this plugin skeleton and the in-development llm changes first, then implement the command, API compatibility layer, and tests against that local checkout.

Explored 4 files, ran 3 commands

• Ran pwd && rg --files -g 'AGENTS.md' -g '!**/.git/**' -g '*' | sed -n '1,240p'

Explored 4 files

• Read ./pyproject.toml
• Read ./llm_chat_completions_server.py
• Read ./tests/test_chat_completions_server.py
• Read ./README.md

• Ran git status --short && git log --oneline --decorate -5 && git diff --stat && git diff

• Ran git status --short && git branch --show-current && git diff main --stat && git diff main -- llm | sed -n '1,320p' && git diff main -- | sed -n '1,1200p'

The repository is still the cookiecutter skeleton, with only the local editable llm source already configured. The relevant llm branch changes are concentrated in message-based conversation handling and async response behavior, so I’m tracing those interfaces before choosing the request/response mapping.

Explored 2 files, a search, ran 2 commands

Explored 4 files, a search

• Read ./llm/parts.py
• Searched for AsyncResponse|async prompt|execute\(|ToolCallPart|ToolResultPart|Message in tests
• Read ./llm/models.py
• Read ./llm/models.py
• Read ./llm/models.py

• Ran git diff main -- llm/models.py && rg -n "class Async|async def|def prompt|def conversation|loaded_messages|_messages_now|Message\(" llm/models.py llm/__init__.py llm/parts.py | sed -n '1,360p'

• Ran rg -n "register_commands|@cli.command|uvicorn|starlette|aiohttp|httpx|serve" . -g '*.py' -g 'pyproject.toml' -g '!**/.venv/**' && uv run llm --help

it should only serve models that are available as async models - it should not require an API token, it should serve /v1/models as a list of available models

Explored 11 files, 2 searches

• Searched for @llm.hookimpl\ndef register_commands|def register_commands in ecosystem
• Read ./llm/models.py
• Read ./llm/models.py
• Read ./llm/models.py
• Read ./llm/models.py
• Read ./llm/models.py
• Read ./llm/parts.py
• Read ./llm/__init__.py
• Searched for def get_(async_)?model|def get_models|AsyncModel in __init__.py
• Read ./tests/test_async.py
• Read ./tests/conftest.py
• Read ./tests/test_openai_messages.py
• Read ./tests/test_openai_messages.py

Got it: the server will be unauthenticated, /v1/chat/completions will reject sync-only models, and /v1/models will enumerate only registered async models. I’ll make model selection go through llm.get_async_model() so aliases still resolve, while the listing reports concrete async model IDs.

Explored 3 files, a search, ran a command

Explored 4 files, a search

• Read ./llm/parts.py
• Read ./llm/__init__.py
• Read ./llm/__init__.py
• Read ./llm/models.py
• Searched for def register_commands[\s\S]{0,1000}@cli\.command in ecosystem

• Ran uv tree --depth 2 && uv run python -c 'import llm; print(llm.__file__); print([(m.model_id, type(m).__name__) for m in llm.get_async_models()])'

build this with red/green TDD but also do manual tests of it while you are building by running a copy of it on some port - support a --reload option so your dev server reloads as the code changes

Explored 3 files, 2 searches, ran a command

Explored 3 files, 2 searches

• Searched for class Options in openai_models.py
• Read ~/Dropbox/dev/ecosystem/datasette-ports/datasette_ports/__init__.py
• Read ~/Dropbox/dev/ecosystem/llm-python/llm_python.py
• Read ~/Dropbox/dev/ecosystem/llm-hello-world/llm_hello_world/__init__.py
• Searched for def build_messages|message.role|role ==|role in in openai_models.py

• Ran sed -n '1,180p' .github/workflows/test.yml && sed -n '1,180p' .github/workflows/publish.yml && sed -n '1,160p' .gitignore && git show HEAD:pyproject.toml

let me know the URL as soon as you have that /v1/models endpoint up and running so I can see it, do that bit first

Explored 6 files, a search

• Read ./llm/default_plugins/openai_models.py
• Read ./llm/default_plugins/openai_models.py
• Read ./llm/models.py
• Searched for class Tool|Tool\( in tests
• Read ./llm/parts.py
• Read ./llm/models.py
• Read ./llm/models.py

I’m switching the first red/green slice to exactly that: command wiring, --reload, and /v1/models. Once its HTTP test is green I’ll start the reload-enabled server on port 8002 and send you the live URL before continuing with completions.

Created a file, edited 2 files, ran 6 commands

• Wrote ./tests/test_chat_completions_server.py (+90 -0)
• Ran uv run pytest -q
• Wrote ./llm_chat_completions_server.py (+50 -5)
• Wrote ./pyproject.toml (+3 -1)
• Ran uv sync && uv run pytest -q
• Ran uv run llm chat-completions-server --reload -p 8002
• Ran lsof -nP -iTCP:8002 -sTCP:LISTEN || true curl -sS -i http://127.0.0.1:8002/v1/models | sed -n '1,80p'
• Ran uv run llm chat-completions-server --reload -p 8003
• Ran curl -sS -i http://127.0.0.1:8003/v1/models | sed -n '1,45p'

/v1/models is live now at http://127.0.0.1:8003/v1/models. Port 8002 was already occupied by a Datasette process, so I used 8003; the server is running with --reload and the endpoint has been manually verified with a 200 response. I’m continuing with the chat-completions endpoint against this same process.

Edited a file, explored 2 files, a search, ran a command

Explored 2 files, a search

• Read ./llm/models.py
• Read ./llm/parts.py
• Searched for class SharedOptions|tool_choice|parallel_tool_calls|class Chat in openai_models.py

• Wrote ./tests/test_chat_completions_server.py (+340 -0)

• Ran uv run pytest -q

OK restart that server adding uv run --with-editable . --with-editable ~/dev/llm --with-editable ~/dev/ecosystem/llm-gemini --with-editable ~/dev/ecosystem/llm-anthropic so we get some extra model plugins

The completion tests are now red for the intended reason (the route does not exist yet). The new tests exercise unauthenticated calls, exact llm.Message translation—including images and tool history—async-only lookup, options, tool definitions/results, non-streaming output, SSE streaming, usage, and OpenAI-shaped errors.

Edited a file

• Wrote ./llm_chat_completions_server.py (+623 -2)

I’ll restart the live reload server with those four editable sources now, keeping it on port 8003 so the URL in your browser remains valid.

Ran a command

• Ran uv run --with-editable . --with-editable ~/dev/llm --with-editable ~/dev/ecosystem/llm-gemini --with-editable ~/dev/ecosystem/llm-anthropic llm chat-completions-server --reload -p 8003

suggest a curl command I can run to try it in non-streaming modee

Ran a command

• Ran curl -sS http://127.0.0.1:8003/v1/models | uv run python -c 'import json,sys; print("\n".join(item["id"] for item in json.load(sys.stdin)["data"] if "gemini" in item["id"].lower() or "claude" in item["id"].lower()))' | sed -n '1,80p'

Plan

• Inspect repository structure and local llm changes versus main
• Implement the chat-completions-server command and API
• Add tests and documentation
• Run focused and full verification

curl -s http://127.0.0.1:8003/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gemini/gemini-2.5-flash",
    "messages": [
      {"role": "user", "content": "Tell me a short joke about pelicans"}
    ],
    "stream": false
  }' | jq

No Authorization header is needed.

It should log to the database, does it not do that yet?

18 previous messages

Correct—it does not log yet. The server currently consumes AsyncResponse and formats the HTTP response, but never calls LLM’s logging path. I’ll add logging after each completed non-streaming response and after a streaming response finishes, using LLM’s normal logs database and the async-to-sync conversion from the in-development branch.

Edited 3 files, explored 2 files, 2 searches, ran 4 commands

Explored 2 files, 2 searches

• Read ./llm/models.py
• Read ./llm/models.py
• Searched for log_to_db\(|logs_db|logging_enabled|log_response in cli.py
• Searched for def get_logs|logs\.db|user_dir\(\).*logs in llm

• Ran uv run pytest -q

Explored 4 files

• Read ./llm/cli.py
• Read ./llm/cli.py
• Read ./llm/cli.py
• Read ./llm/models.py

• Wrote ./tests/test_chat_completions_server.py (+64 -5)

• Ran uv run pytest -q

• Wrote ./llm_chat_completions_server.py (+15 -0)

• Wrote ./pyproject.toml (+1 -0)

• Wrote ./llm_chat_completions_server.py (+1 -0)

• Ran uv sync && uv run pytest -q

• Wrote ./tests/test_chat_completions_server.py (+4 -1)

• Ran uv run pytest -q

Add it using the very latets code in the current ~/dev/llm branch

Logging is now implemented and both non-streaming and streaming database tests are green. I’m checking it against the live Gemini-backed server and the actual default LLM logs database now.

Ran a command

• Ran curl -sS http://127.0.0.1:8003/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"gemini/gemini-2.5-flash","messages":[{"role":"user","content":"Reply with exactly: logging works"}],"stream":false}' | jq '{id, model, choices, usage, error}' uv run llm logs list --help | sed -n '1,180p'

Understood—I’ll target the current working tree in ~/dev/llm, not the released package API. The live server already has that checkout injected with --with-editable ~/dev/llm; I’m re-reading its latest response/logging interfaces now in case they changed while we were building.

Ran 3 commands

• Ran realpath ~/dev/llm && git branch --show-current && git rev-parse HEAD && git status --short && git diff main -- llm/models.py llm/logs.py | sed -n '1,520p'
• Ran uv run --with-editable . --with-editable ~/dev/llm --with-editable ~/dev/ecosystem/llm-gemini --with-editable ~/dev/ecosystem/llm-anthropic python -c 'import inspect,llm; print(llm.__file__); print(hasattr(llm.AsyncResponse, "log_to_db")); print(inspect.getsource(llm.AsyncResponse.to_sync_response))'
• Ran uv run llm logs list -n 1 --json

Confirmed: ~/dev/llm resolves to /Users/simon/Dropbox/dev/llm, currently on content-addressed-logs at 237f047…, and the live process imports LLM directly from that checkout. I’m using that branch’s own async logging flow—await AsyncResponse.to_sync_response(), then log_to_db()—which writes both the legacy response tables and the new content-addressed messages/turns tables. The live Gemini request is present in llm logs with its structured messages and reasoning metadata.

Ran a command

• Ran curl -sS -N http://127.0.0.1:8003/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"gemini/gemini-2.5-flash","messages":[{"role":"system","content":"Be concise."},{"role":"user","content":"Reply with exactly two words: stream works"}],"stream":true,"stream_options":{"include_usage":true}}' | sed -n '1,24p' uv run llm logs list -n 1 --json | uv run python -c 'import json,sys; row=json.load(sys.stdin)[0]; print({key: row[key] for key in ("id", "model", "response", "input_tokens", "output_tokens")})'

Restart the server with an extra plugin --with-editable ~/dev/ecosystem/llm-apple-foundation

I’ll restart it on the same port with the Apple Foundation Models plugin added, keeping all four existing editable sources and --reload.

Ran 2 commands

• Ran uv run --with-editable . --with-editable ~/dev/llm --with-editable ~/dev/ecosystem/llm-gemini --with-editable ~/dev/ecosystem/llm-anthropic --with-editable ~/dev/ecosystem/llm-apple-foundation llm chat-completions-server --reload -p 8003
• Ran curl -sS http://127.0.0.1:8003/v1/models | uv run python -c 'import json,sys; ids=[item["id"] for item in json.load(sys.stdin)["data"]]; print("\n".join(model_id for model_id in ids if "apple" in model_id.lower() or "foundation" in model_id.lower())); print(f"total async models: {len(ids)}")'

Restarted successfully. http://127.0.0.1:8003/v1/models is live again and now lists apple-foundation among 113 async models.

The live server is stable with the added plugin. I’m finishing the package handoff now: documenting the endpoints, async-only behavior, no-auth contract, logging semantics, reload workflow, and both streaming modes, then I’ll run the full verification once more.

Edited 2 files, ran 5 commands

• Wrote ./README.md (+86 -4)
• Ran git status --short && git diff --stat && git diff -- pyproject.toml README.md llm_chat_completions_server.py tests/test_chat_completions_server.py | sed -n '1,1400p'
• Ran uvx ruff check llm_chat_completions_server.py tests && uvx ruff format --check llm_chat_completions_server.py tests
• Ran uv run llm chat-completions-server --help && uv run llm plugins
• Wrote ./README.md (+1 -0)
• Wrote ./llm_chat_completions_server.py (+0 -1)
• Ran uvx ruff format llm_chat_completions_server.py tests/test_chat_completions_server.py uvx ruff check llm_chat_completions_server.py tests uv run pytest -q
• Ran curl -sS http://127.0.0.1:8003/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"apple-foundation","messages":[{"role":"user","content":"Reply with exactly: apple works"}],"stream":false}' | jq '{id, model, choices, usage, error}'

The Apple plugin is loaded and the endpoint completed a real apple-foundation request successfully. I’m doing one last compatibility pass through the official OpenAI Python client, plus package build and test checks.

Ran 4 commands

• Ran uv run python -c 'from openai import OpenAI; c=OpenAI(base_url="http://127.0.0.1:8003/v1", api_key="ignored"); r=c.chat.completions.create(model="apple-foundation", messages=[{"role":"user","content":"Say hello in one word"}]); print(type(r).__name__, r.model, repr(r.choices[0].message.content), r.choices[0].finish_reason)'
• Ran uv build
• Ran git diff --check && uv lock --check && uv run pytest -q && uvx ruff check llm_chat_completions_server.py tests && uvx ruff format --check llm_chat_completions_server.py tests
• Ran git status --short && curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8003/v1/models && uv run llm logs list -n 1 --json | uv run python -c 'import json,sys; row=json.load(sys.stdin)[0]; print(row["model"], row["id"])'

Plan

Documentation is complete, and unit, lint, formatting, lockfile, package-build, live curl, database-log, and OpenAI-client compatibility checks all passed.

• Inspect repository structure and local llm changes versus main
• Implement the chat-completions-server command and API
• Add tests and documentation
• Run focused and full verification

Implemented and live at http://127.0.0.1:8003/v1/models.

Key results:

• llm chat-completions-server, default port 8002, with -p/--port and --reload
• Async-only /v1/models
• Unauthenticated /v1/chat/completions
• Streaming SSE and non-streaming responses
• Messages, images, tools, options, schemas, and usage
• Logging to legacy and content-addressed tables using the latest ~/dev/llm
• Live server includes Gemini, Anthropic, and Apple Foundation plugins
• 10 tests passing; lint, formatting, lockfile, package build, curl, and OpenAI Python client verified

Files: implementation, tests, documentation.

weird, look at the most recently logged message (uv run llm logs -c) - the prompt is missing

4 previous messages

I’ll inspect the latest legacy log row and its content-addressed message chain. My suspicion is that explicit messages= are stored correctly in the new tables but the legacy prompt column remains null because the server calls model.prompt(messages=...) without the legacy prompt= argument.

Ran 3 commands

• Ran uv run llm logs list -n 1 --json
• Ran uv run llm logs -c
• Ran uv run python -c 'import llm, sqlite_utils, json; db=sqlite_utils.Database(llm.user_dir()/"logs.db"); r=next(db.query("select * from responses order by id desc limit 1")); print("response", {k:r[k] for k in ("id","prompt","system","conversation_id","response")}); t=next(db.query("select * from turns order by id desc limit 1")); print("turn", t); from llm.logs import LogStore; print(json.dumps([m.to_dict() for m in LogStore(db).load_chain(t["tip_message_hash"])], indent=2))'

Confirmed: this is a compatibility gap between the two logging schemas. The content-addressed chain has the full user prompt, but the legacy responses.prompt column is null, so llm logs -c renders -- none --. I’ll populate the legacy prompt and system fields while continuing to pass the full canonical messages= chain to the async model.

Edited 2 files, ran 3 commands

• Wrote ./tests/test_chat_completions_server.py (+12 -3)
• Ran uv run pytest -q tests/test_chat_completions_server.py::test_completed_responses_are_logged
• Wrote ./llm_chat_completions_server.py (+17 -0)
• Ran uv run pytest -q tests/test_chat_completions_server.py::test_completed_responses_are_logged && uv run pytest -q && uvx ruff check llm_chat_completions_server.py tests && uvx ruff format --check llm_chat_completions_server.py tests
• Ran curl -sS http://127.0.0.1:8003/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"gemini/gemini-2.5-flash","messages":[{"role":"system","content":"Be concise"},{"role":"user","content":"Reply with exactly: prompt restored"}],"stream":false}' | jq -r '.choices[0].message.content // .error.message' uv run llm logs -c | sed -n '1,18p'

Fixed. The full messages existed in the content-addressed tables, but legacy responses.prompt and responses.system were null.

The server now populates those legacy fields while retaining the canonical messages= chain. Verified live with uv run llm logs -c; it now displays:

• Prompt: Reply with exactly: prompt restored
• System: Be concise

All 10 tests pass, and the server has reloaded on port 8003.