Skip to content

Instantly share code, notes, and snippets.

@simonw

simonw/gemini-live.md Secret

Last active September 16, 2026 00:10
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save simonw/067b7430c5b1f743af9419b0184c38ef to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save simonw/067b7430c5b1f743af9419b0184c38ef to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gemini-live.md

In ~/dev/tools any tools that use Gemini live audio ?

In ~/dev/tools any tools that use Gemini live audio ?

2 previous messages

I’ll check ~/dev/tools for Gemini Live audio integrations.

Explored 2 files, 6 searches, ran 2 commands

  • Ran pwd && rg --files -g AGENTS.md -g '*gemini*' -g '*Gemini*' -g '*live*' -g '*audio*' -g '!node_modules' -g '!.git' -g '!.venv' /Users/simon/dev/tools

  • Ran for p in /AGENTS.md /Users/AGENTS.md /Users/simon/AGENTS.md /Users/simon/dev/AGENTS.md /Users/simon/dev/tools/AGENTS.md; do if [ -f "$p" ]; then cat "$p"; fi; done

Explored 2 files, 6 searches
  • Searched for gemini.{0,100}(live|audio)|(live|audio).{0,100}gemini|BidiGenerateContent|bidiGenerateContent|sendRealtimeInput|realtimeInput|live\.connect|native.audio in .
  • Read ./gemini-flash-tts.docs.md
  • Searched for model|generateContent|responseModalities|AUDIO|fetch\( in gemini-flash-tts.html
  • Searched for audio|live|realtime in gemini-chat.html
  • Searched for generativelanguage|@google/genai|@google/generative|aiplatform|WebSocket|gemini- in .
  • Searched for 240 in .connect|native.audio
  • Read ./openai-webrtc.docs.md
  • Searched for api.openai.com|realtime|RTCPeerConnection|getUserMedia in openai-webrtc.html

I found no Gemini Live audio integrations in ~/dev/tools.

Closest tools:

  • gemini-flash-tts.html — Gemini text-to-speech using generateContent, with single/multiple speakers and WAV downloads.
  • openai-webrtc.html — live audio conversations using OpenAI’s Realtime API over WebRTC.

Read https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-live-gemini-3-8-live-extended-thinking/ and associated documentation and build a new tool

12 previous messages

I’ll read the announcement and API docs, then build a Gemini Live audio tool in ~/dev/tools using the repository’s conventions.

Explored 3 files, a list, ran 3 commands, searched the web
Explored a file
  • Read ~/.codex/plugins/cache/openai-bundled/sites/0.1.57/skills/sites-building/SKILL.md
  • Ran git status --short && rg --files -g 'AGENTS.md' -g '*README*' -g '*test*' -g '*package*' -g '.openai/**' -g '.github/**' -g '*CONTRIBUTING*' -g '*CLAUDE*' -g '*GEMINI*' -g '!node_modules' && sed -n '1,220p' README.md && sed -n '1,240p' openai-webrtc.html

Searched the web for https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-live-gemini-3-8-live-extended-thinking/

  • Ran curl -L --fail --max-time 30 'https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-live-gemini-3-8-live-extended-thinking/' -o /tmp/gemini-live-announcement.html && wc -c /tmp/gemini-live-announcement.html && cat package.json && cat tests/conftest.py && cat tests/test_video_compressor.py && sed -n '240,480p' openai-webrtc.html && cat .gitignore

Searched the web for site:blog.google "Gemini 3.8 Live" ...

  • Ran uv run --with beautifulsoup4 python - <<'PY' from bs4 import BeautifulSoup from pathlib import Path p = Path('/tmp/gemini-live-announcement.html') soup = BeautifulSoup(p.read_text(), 'html.parser') body = soup.find('main') or soup print(body.get_text('\n', strip=True)) print('\nDOCUMENTATION LINKS') for a in body.find_all('a', href=True): if any(x in a['href'] for x in ['ai.google.dev', 'github', 'cloud.google', 'aistudio']): print(a.get_text(' ', strip=True), a['href']) PY
Explored 3 files, a list
  • Read ./package.json
  • Read ./tests/conftest.py
  • Read ./pyproject.toml
  • Listed files

The two models need different state handling: Extended Thinking can keep working after a spoken response ends. I’ll support both models, with live transcripts, interruption handling, voice selection, and optional document context.

Explored a file, a search, ran 4 commands, searched the web

Searched the web for https://ai.google.dev/gemini-api/docs/live-api

  • Ran uv run --with beautifulsoup4 python - <<'PY' from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen from bs4 import BeautifulSoup from pathlib import Path urls = {'overview':'https://ai.google.dev/gemini-api/docs/live-api', 'thinking':'https://ai.google.dev/gemini-api/docs/live-api/thinking', 'reference':'https://ai.google.dev/api/live', 'capabilities':'https://ai.google.dev/gemini-api/docs/live-guide', 'session':'https://ai.google.dev/gemini-api/docs/live-session'} def get(item): name, url = item soup=BeautifulSoup(urlopen(url).read(), 'html.parser') main=soup.find('article') or soup.find('main') or soup Path('/tmp/gemini-live-'+name+'.txt').write_text(main.get_text('\n', strip=True)) links=[(a.get_text(' ',strip=True),a['href']) for a in main.find_all('a',href=True) if any(s in a['href'] for s in ['live','github'])] return name,links for result in ThreadPoolExecutor().map(get, urls.items()): print(result) PY
Explored a file, a search
  • Read /tmp/gemini-live-thinking.txt
  • Searched for audioStreamEnd|voice|Voice|interrupted|inputAudioTranscription|outputAudioTranscription|promptTokenCount|contextWindowCompression in gemini-live-capabilities.txt

  • Ran uv run --with beautifulsoup4 python - <<'PY' from urllib.request import urlopen from bs4 import BeautifulSoup soup=BeautifulSoup(urlopen('https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket').read(),'html.parser') print((soup.find('article') or soup).get_text('\n',strip=True)) PY

  • Ran uv run --with beautifulsoup4 python - <<'PY' from urllib.request import urlopen from bs4 import BeautifulSoup for name,url,keys in [('capabilities','https://ai.google.dev/gemini-api/docs/live-guide',['voice','audio transcription','interrupt','token count']),('reference','https://ai.google.dev/api/live',['UsageMetadata','Transcription','BidiGenerateContentSetup','SessionResumptionUpdate'])]: soup=BeautifulSoup(urlopen(url).read(),'html.parser') for h in soup.select('h2,h3'): if any(k.lower() in h.get_text(' ',strip=True).lower() for k in keys): print('\n##',h.get_text(' ',strip=True)) for sib in h.next_siblings: if getattr(sib,'name',None) in ['h2','h3']: break if getattr(sib,'name',None): print(sib.get_text(' ',strip=True)[:5500]) PY

  • Ran rg -n 'localStorage|voiceNames|Puck|Kore' gemini-flash-tts.html gemini-chat.html && command -v node && ls node_modules/@playwright/test/package.json

I’ve checked the microphone format, playback format, and session protocol. I’m building a single HTML file to match the other tools, with direct browser-to-Google audio, a mute control, and explicit status for background thinking.

Created 2 files, ran 2 commands
  • Wrote ~/dev/tools/gemini-live.docs.md (+14 -0)
  • Wrote ~/dev/tools/gemini-live.html (+501 -0)
  • Ran uv run python - <<'PY' import inspect from playwright.sync_api import WebSocketRoute print(inspect.signature(WebSocketRoute.close)) print(inspect.signature(WebSocketRoute.on_message)) PY
  • Ran node --input-type=module - <<'JS' import fs from 'node:fs'; import vm from 'node:vm'; const html = fs.readFileSync('gemini-live.html', 'utf8'); const source = html.match(/<script type="module">([\s\S]*?)<\/script>/)[1]; new vm.Script(source); console.log('JavaScript syntax OK'); JS

The tool is implemented. I’m testing the audio stream and session controls, especially interruption playback, microphone cleanup, reconnection, and Extended Thinking staying active between spoken updates.

Created a file, edited a file, ran 5 commands
  • Wrote ~/dev/tools/tests/test_gemini_live.py (+236 -0)
  • Ran uv run pytest tests/test_gemini_live.py -q
  • Wrote ~/dev/tools/tests/test_gemini_live.py (+1 -1)
  • Ran uv run pytest tests/test_gemini_live.py -q
  • Ran git diff --check && git status --short && rg --files .github | head -20
  • Wrote ~/dev/tools/tests/test_gemini_live.py (+29 -3)
  • Ran tail -90 tests/test_gemini_live.py && uv run pytest tests/test_gemini_live.py -q
  • Wrote ~/dev/tools/tests/test_gemini_live.py (+8 -8)
  • Ran uv run pytest tests/test_gemini_live.py -q

All 11 tests passed using a simulated Gemini connection and real browser audio processing. I’m checking the final page layout and opening a local preview. A live conversation still needs your Gemini API key.

Edited a file, ran a command

  • Wrote ~/dev/tools/gemini-live.html (+1 -1)

  • Ran uv run python - <<'PY' from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page(viewport={'width': 1200, 'height': 1100}, device_scale_factor=1) page.goto('http://localhost:49418/gemini-live.html') page.screenshot(path='/tmp/gemini-live-desktop.png', full_page=True) page.set_viewport_size({'width': 390, 'height': 844}) page.select_option('#model', 'gemini-3.8-live-extended-thinking') page.screenshot(path='/tmp/gemini-live-mobile.png', full_page=True) print('Desktop and mobile screenshots saved.') browser.close() PY

Tool call

Tool: open_in_codex Status: completed

Viewed 2 images

Ran 4 commands
  • Ran uv run pytest tests/test_gemini_live.py -q && git diff --no-index --check /dev/null gemini-live.html && git diff --no-index --check /dev/null gemini-live.docs.md && git diff --no-index --check /dev/null tests/test_gemini_live.py && git status --short
  • Ran git diff --no-index --check /dev/null gemini-live.docs.md
  • Ran git diff --no-index --check /dev/null tests/test_gemini_live.py
  • Ran git status --short

Built gemini-live.html. Try the local preview.

Includes both Gemini 3.8 Live models, thinking levels, voice selection, live transcripts, interruptions, document context, and session reconnection.

11 tests pass. Live API testing still requires your Gemini key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment