Watcher Thread Notification Failure
| Metric | Value |
|---|---|
| Bugs Identified | 7 (2 Critical, 2 High, 3 Medium) |
| Root Cause | Missing WebSocket broadcast in add_message() |
| Fix Complexity | ~5 lines of code |
| Components Affected | Workflows, Autopilots, Thread Messaging |
| Impact | Users must refresh to see workflow results |
┌─────────────────────────────────────────────────────────────────┐
│ April 16, 2026 │
│ └─ Initial ActRun production audit │
│ ├─ 20+ routes mapped │
│ ├─ 13 integrations tested │
│ └─ 4 bugs found (Gmail 404, auth wipe, etc.) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ April 23, 2026 — Telegram Audit │
│ └─ Customer-reported issue: Telegram bot requires Gmail │
│ ├─ Created @actrun_agent_bot for testing │
│ ├─ Reproduced email dependency bug ✅ │
│ ├─ Discovered ghost message bug (~50% failure rate) │
│ ├─ Found UI staleness bug (no refresh after connect) │
│ └─ Found silent bot replacement bug │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ April 23, 2026 — Workflow/Watcher Investigation │
│ └─ Static code analysis from coding agent │
│ ├─ Identified `add_message()` missing WebSocket broadcast │
│ ├─ Confirmed asymmetry with `modify_message()` │
│ └─ Documented full fix and testing plan │
└─────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ TERMINOLOGY │
├────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Autopilots │ │ Workflows │ │ Watcher │ │
│ │ (Sidebar) │────▶│ (Sidebar) │────▶│ Thread │ │
│ │ │ │ │ │ (/chat/{id}) │ │
│ │ = │ │ = │ │ = │ │
│ │ WatcherAgent │ │ AutomatedJob │ │ EmailThread │ │
│ │ records │ │ records │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ │ owns │ contains │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Key Relationship │ │
│ │ watcher_agents.thread_id → email_threads.thread_id │ │
│ │ automated_jobs.watcher_id → watcher_agents.id │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ EXPECTED FLOW (Working) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Job Completes post_result_to_watcher_thread() │
│ ──────────── ─────────────▶ ┌─────────────────────────────┐ │
│ │ Blaxel │ │ Format summary │ │
│ │ Worker │ │ + status emoji │ │
│ └───────────┘ │ + result file (truncated) │ │
│ │ + token/cost info │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ▼ │
│ message_handler.add_message() │
│ ┌─────────────────────────────┐ │
│ │ Write to DB │ │
│ │ email_thread_messages │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ▼ │
│ websocket_manager.send_ │
│ thread_update(thread_id) │
│ ┌─────────────────────────────┐ │
│ │ Broadcast via Redis pub/sub│ │
│ │ → All backend instances │ │
│ │ → All connected clients │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────┐ │
│ │ Frontend WebSocket Receives │ │
│ │ threads-provider.tsx │ │
│ │ → UI updates in real-time! │ │
│ └─────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ ACTUAL FLOW (Broken) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Job Completes post_result_to_watcher_thread() │
│ ──────────── ─────────────▶ ┌─────────────────────────────┐ │
│ │ Blaxel │ │ Format summary │ │
│ │ Worker │ │ + status emoji │ │
│ └───────────┘ │ + result file (truncated) │ │
│ │ + token/cost info │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ▼ │
│ message_handler.add_message() │
│ ┌─────────────────────────────┐ │
│ │ Write to DB │ │
│ │ email_thread_messages │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ▼ │
│ ❌ MISSING: send_thread_update() │
│ ┌─────────────────────────────┐ │
│ │ No WebSocket broadcast │ │
│ │ → No Redis pub/sub │ │
│ │ → Frontend never notified │ │
│ └─────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────┐ │
│ │ User must refresh page │ │
│ │ → REST fetch loads messages │ │
│ │ → Result finally appears │ │
│ └─────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ 🔴 CRITICAL (Fix Immediately) │
├────────────────────────────────────────────────────────────────────┤
│ 1. Ghost Messages — Bot ignores ~50% of messages │
│ └─ User message appears → No response → "No messages here..."│
│ │
│ 2. UI Stale State — Connect/disconnect doesn't refresh │
│ └─ API operation succeeds → UI shows old data → Manual reload │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ 🟡 HIGH (Before Customer Rollout) │
├────────────────────────────────────────────────────────────────────┤
│ 3. Silent Bot Replacement — New token replaces old without warning│
│ └─ connect() with new token → Old bot instantly disconnected │
│ │
│ 4. Email Dependency — Bot needs Gmail for email tasks │
│ └─ "Email access disconnected" error blocks functionality │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ 🟢 MEDIUM (Post-Launch) │
├────────────────────────────────────────────────────────────────────┤
│ 5. Orphaned Integrations — Each connect creates new record │
│ └─ connect() → disconnect() → connect() = 3 DB records │
│ │
│ 6. Missing API Endpoints — /test, /send, /webhook return 404 │
│ │
│ 7. API Prefix Inconsistency — Mixed /integrations/ vs /api/ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ 🔴 HIGH (Core Feature Broken) │
├────────────────────────────────────────────────────────────────────┤
│ 8. WebSocket Broadcast Missing — Results don't appear in real-time│
│ └─ add_message() writes to DB → No WebSocket → No UI update │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 1: Fix Ghost Message Bug │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Investigate message processing pipeline │ │
│ │ → Check for race conditions / timeouts │ │
│ │ → Add logging for message deduplication │ │
│ │ → Test with different message types │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Expected Impact: │ │
│ │ - Bot becomes reliable (100% response rate) │ │
│ │ - User trust restored │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 2: Fix UI Stale State │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Add WebSocket/Reactive updates after connect/disconnect │ │
│ │ → Implement polling fallback if WS unavailable │ │
│ │ → Add visual loading indicators │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Expected Impact: │ │
│ │ - Users see current integration state immediately │ │
│ │ - Reduces confusion after config changes │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 3: Add Bot Replacement Warning │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Check for existing bot before connecting new one │ │
│ │ → Return 409 Conflict if bot exists │ │
│ │ → Or require ?force=true parameter │ │
│ │ → Add confirmation modal in UI │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 4: Fix WebSocket Broadcast (Watcher Notification) │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Add send_thread_update() to add_message() │ │
│ │ → Mirror modify_message() behavior │ │
│ │ → Test with multi-instance Redis pub/sub │ │
│ │ → ~5 lines of code │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Expected Impact: │ │
│ │ - Watcher threads update in real-time │ │
│ │ - Workflow results appear immediately │ │
│ │ - Autopilot experience much smoother │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 5: Improve Email Dependency Error Handling │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Make bot functional for non-email tasks without Gmail │ │
│ │ → Clearer error messaging │ │
│ │ → Suggest connecting Gmail when needed │ │
│ │ → Show which features require email vs. don't │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 6: Fix Disconnect Cleanup │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Properly deactivate old integrations │ │
│ │ → Don't create duplicate records │ │
│ │ → Clean up webhooks │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 7: Simplify Setup Flow │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Add "Connect Telegram" button with token input │ │
│ │ → Step-by-step BotFather instructions │ │
│ │ → Auto-verify webhook after connect │ │
│ │ → Success confirmation with bot details │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 8: Standardize API Prefixes │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Choose one pattern: /integrations/{service}/ │ │
│ │ → Migrate all endpoints to consistent pattern │ │
│ │ → Update documentation │ │
│ │ → Fix Gmail endpoint (currently 404 on all paths) │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────┐
│ Priority 9: Add Skill Suggestions │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Preload/suggest skills during autopilot setup │ │
│ │ → Categorize skills (Email, Research, Content, etc.) │ │
│ │ → Show available skills based on connected integrations │ │
│ │ → One-click enable/disable │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
Location: core/agent/handlers/message_handler.py
async def add_message(
thread_id: str,
content: str,
role: str,
files: Optional[List[str]] = None,
metadata: Optional[dict] = None,
) -> EmailThreadMessage:
# ... existing database write code ...
# ✅ ADD THIS LINE:
await websocket_manager.send_thread_update(thread_id)
return messagePros:
- Simple (~5 lines of code)
- Mirrors existing
modify_message()behavior - Fixes the immediate issue
- Low risk
Cons:
- Broadcasts ALL message additions (may be unnecessary for some use cases)
- No filtering based on message type/role
Effort: 2-4 hours (including testing)
async def add_message(..., should_broadcast: bool = True):
# ... existing code ...
if should_broadcast:
await websocket_manager.send_thread_update(thread_id)
return messagePros:
- More flexibility for callers
- Can skip broadcast for internal/system messages
Cons:
- More complex API surface
- Requires updating all callers
- Easy to forget to set
should_broadcast=True
Effort: 4-6 hours (including updating callers)
async def add_message(...):
# ... existing code ...
if role == "system":
await websocket_manager.send_thread_update(thread_id)
return messagePros:
- Only broadcasts system messages (workflow results, etc.)
- Skips broadcast for user messages (handled elsewhere)
Cons:
- Watcher's assistant response uses
role="assistant"— wouldn't broadcast - Doesn't fix the Watcher auto-response issue
- Harder to reason about
Effort: 3-5 hours (but doesn't fully fix the bug)
async def add_message(...):
message = await db_write(...)
# Unified broadcast logic
await websocket_manager.send_thread_update(thread_id)
return messageAnd remove duplicate broadcast from modify_message():
async def modify_message(...):
message = await db_update(...)
# ❌ REMOVE: await websocket_manager.send_thread_update(thread_id)
# (Now handled in add_message/modify_message unified path)
return messagePros:
- Cleanest architecture
- No duplication
- Broadcasts all updates consistently
Cons:
- Requires refactoring
modify_message()callers - Risk of breaking existing code
- May introduce bugs during refactoring
Effort: 8-12 hours (high risk)
Implementation:
# core/agent/handlers/message_handler.py
async def add_message(
thread_id: str,
content: str,
role: str,
files: Optional[List[str]] = None,
metadata: Optional[dict] = None,
) -> EmailThreadMessage:
"""
Add a message to a thread and broadcast WebSocket update.
Args:
thread_id: The thread ID to add the message to
content: The message content
role: Message role (user, assistant, system, etc.)
files: Optional list of file IDs
metadata: Optional metadata dictionary
Returns:
The created EmailThreadMessage
"""
# Write to database
message = await db.create_email_thread_message(
thread_id=thread_id,
content=content,
role=role,
files=files or [],
metadata=metadata or {}
)
# Broadcast WebSocket update to all connected clients
# This mirrors the behavior in modify_message() and modify_message_by_id()
try:
await websocket_manager.send_thread_update(thread_id)
except Exception as e:
# Log but don't fail - message is already in DB
logger.error(f"Failed to broadcast thread update: {e}")
return messageWhy This Approach:
- Simple & Low Risk — Minimal code change, follows existing pattern
- Consistent — Mirrors
modify_message()behavior exactly - Robust — Error handling ensures broadcast failures don't break message creation
- Well-Documented — Clear docstring explaining the behavior
- Testable — Easy to verify broadcast happens
Complementary Changes:
# Add unit test for WebSocket broadcast
async def test_add_message_broadcasts_ws():
# Create thread
thread = await create_thread()
# Mock WebSocket manager
with patch('websocket_manager.send_thread_update') as mock_broadcast:
# Add message
msg = await add_message(thread_id=thread.id, content="test", role="user")
# Verify broadcast was called
mock_broadcast.assert_called_once_with(thread.id)Rollback Plan:
If broadcast causes issues (e.g., performance, spam):
- Feature flag the broadcast:
if config.BROADCAST_MESSAGES: await ... - Or add rate limiting:
if not rate_limiter.is_limited(thread_id): await ...
User Experience:
├─ Open watcher thread
├─ Trigger workflow
├─ Wait 10-60 seconds
├─ Nothing happens...
├─ Refresh page
└─ Result appears 😤
Technical Impact:
├─ Messages written to DB ✅
├─ WebSocket never fires ❌
├─ Frontend never updates ❌
└─ Users think system is broken 😞
User Experience:
├─ Open watcher thread
├─ Trigger workflow
├─ Wait 10-60 seconds
├─ ✨ Result appears automatically! ✨
└─ Happy user 😊
Technical Impact:
├─ Messages written to DB ✅
├─ WebSocket fires ✅
├─ Frontend updates in real-time ✅
└─ Users see system working as intended 🎉
| Metric | Before | After | Impact |
|---|---|---|---|
| DB writes | 1 per message | 1 per message | No change |
| WebSocket broadcasts | 0 per message | 1 per message | +1 Redis pub/sub |
| Client updates | On refresh only | Real-time | Better UX |
| Network traffic | No WS traffic | ~1KB/message | Negligible |
Conclusion: Minimal performance impact, significant UX improvement.
# Test 1: Broadcast happens on add_message
assert mock_send_thread_update.called_with(thread_id)
# Test 2: Broadcast doesn't fail if WS manager errors
# (should log error but still return message)
# Test 3: Multiple messages all broadcast
for i in range(5):
await add_message(...)
assert mock_send_thread_update.call_count == 5# Test 1: Real-time update in browser
# - Open thread in browser (WS connected)
# - Trigger workflow via API
# - Assert message appears without refresh
# Test 2: Multiple tabs update
# - Open same thread in 3 tabs
# - Add message via API
# - Assert all 3 tabs update
# Test 3: Multi-instance Redis pub/sub
# - Backend instance A writes message
# - Backend instance B's client should receive broadcast- Open watcher thread in browser
- Trigger workflow owned by watcher
- Confirm message appears in real-time (no refresh)
- Check browser console for WebSocket errors
- Test with multiple browser tabs
- Test Watcher's auto-response also broadcasts
- Verify no performance degradation
- Check Redis pub/sub logs
- Engineering: Implement Option 1 fix in
message_handler.py - Engineering: Add unit tests for WebSocket broadcast
- QA: Set up test environment with Redis
- QA: Execute integration tests
- Engineering: Code review
- Engineering: Deploy fix to staging
- QA: Run full regression test suite
- Engineering: Fix ghost message bug (Telegram)
- Engineering: Fix UI stale state bug
- Engineering: Add bot replacement warning
- Engineering: Standardize API prefixes
- Product: Simplify Telegram setup flow
- Engineering: Implement skill suggestions
- QA: Full end-to-end testing of all fixes
| File | Purpose | Lines of Interest |
|---|---|---|
core/agent/handlers/message_handler.py |
Contains bug | add_message() function |
jobs/tasks.py |
Calls buggy function | 387-640 |
core/automation/routes.py |
Manual trigger endpoint | 361-395 |
core/watcher/routes.py |
Watcher run-now endpoint | 707-746 |
website/src/providers/threads-provider.tsx |
Frontend WS listener | — |
models.py |
Database schemas | watcher_agents (1361), automated_jobs (847), email_threads (797) |
- Bug Report:
~/vic-workspace/actrun-qa/bug-report-watcher-notification.md - Telegram Audit:
~/vic-workspace/actrun-qa/telegram-audit-2026-04-23.md - Original Investigation:
~/vic-workspace/actrun-qa/investigation-2026-04-16.md
- Telegram bot token:
/tmp/actrun-bot-token.txt - Test scripts:
/tmp/ar-tg-*.mjsand/tmp/ar-workflow-*.mjs
- Telegram Ghost Messages — Similar pattern: message created but no response
- UI Stale State — Different manifestation of same root cause (missing updates)
- Watcher Auto-Response — Also uses
add_message(), would be fixed by same change
Created by: Vic (AI chief of staff) Date: April 23, 2026 Status: Ready for Engineering Review