Skip to content

Instantly share code, notes, and snippets.

@philipbankier
Created April 23, 2026 23:00
Show Gist options
  • Select an option

  • Save philipbankier/dd153d17b4a38778820c23c14e8320c8 to your computer and use it in GitHub Desktop.

Select an option

Save philipbankier/dd153d17b4a38778820c23c14e8320c8 to your computer and use it in GitHub Desktop.
ActRun Bug Investigation: Visual Summary - Complete breakdown of 8 bugs, fix options, and comprehensive action plan with diagrams

ActRun Bug Investigation: Visual Summary

Watcher Thread Notification Failure


📊 Executive Summary

Metric Value
Bugs Identified 7 (2 Critical, 2 High, 3 Medium)
Root Cause Missing WebSocket broadcast in add_message()
Fix Complexity ~5 lines of code
Components Affected Workflows, Autopilots, Thread Messaging
Impact Users must refresh to see workflow results

🔍 Investigation Timeline

┌─────────────────────────────────────────────────────────────────┐
│  April 16, 2026                                                 │
│  └─ Initial ActRun production audit                            │
│     ├─ 20+ routes mapped                                        │
│     ├─ 13 integrations tested                                   │
│     └─ 4 bugs found (Gmail 404, auth wipe, etc.)               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  April 23, 2026 — Telegram Audit                               │
│  └─ Customer-reported issue: Telegram bot requires Gmail        │
│     ├─ Created @actrun_agent_bot for testing                   │
│     ├─ Reproduced email dependency bug ✅                      │
│     ├─ Discovered ghost message bug (~50% failure rate)         │
│     ├─ Found UI staleness bug (no refresh after connect)       │
│     └─ Found silent bot replacement bug                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  April 23, 2026 — Workflow/Watcher Investigation               │
│  └─ Static code analysis from coding agent                      │
│     ├─ Identified `add_message()` missing WebSocket broadcast   │
│     ├─ Confirmed asymmetry with `modify_message()`             │
│     └─ Documented full fix and testing plan                   │
└─────────────────────────────────────────────────────────────────┘

🏗️ System Architecture

Component Definitions

┌────────────────────────────────────────────────────────────────────┐
│                        TERMINOLOGY                                │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐    │
│  │  Autopilots  │     │  Workflows   │     │ Watcher      │    │
│  │  (Sidebar)   │────▶│  (Sidebar)   │────▶│ Thread       │    │
│  │              │     │              │     │ (/chat/{id}) │    │
│  │ =            │     │ =            │     │ =            │    │
│  │ WatcherAgent │     │ AutomatedJob │     │ EmailThread  │    │
│  │ records      │     │ records      │     │              │    │
│  └──────────────┘     └──────────────┘     └──────────────┘    │
│         │                                       │                │
│         │ owns                                 │ contains       │
│         ▼                                       ▼                │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │              Key Relationship                          │  │
│  │   watcher_agents.thread_id → email_threads.thread_id   │  │
│  │   automated_jobs.watcher_id → watcher_agents.id        │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

Message Flow (Expected vs. Actual)

┌─────────────────────────────────────────────────────────────────────┐
│                    EXPECTED FLOW (Working)                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Job Completes               post_result_to_watcher_thread()       │
│  ────────────  ─────────────▶  ┌─────────────────────────────┐    │
│  │ Blaxel    │               │ Format summary            │    │
│  │ Worker    │               │ + status emoji             │    │
│  └───────────┘               │ + result file (truncated) │    │
│                              │ + token/cost info         │    │
│                              └──────────┬──────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              message_handler.add_message()      │
│                              ┌─────────────────────────────┐    │
│                              │ Write to DB               │    │
│                              │ email_thread_messages     │    │
│                              └──────────┬──────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              websocket_manager.send_         │
│                              thread_update(thread_id)     │
│                              ┌─────────────────────────────┐    │
│                              │ Broadcast via Redis pub/sub│    │
│                              │ → All backend instances    │    │
│                              │ → All connected clients   │    │
│                              └──────────┬──────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              ┌─────────────────────────────┐    │
│                              │ Frontend WebSocket Receives │    │
│                              │ threads-provider.tsx       │    │
│                              │ → UI updates in real-time! │    │
│                              └─────────────────────────────┘    │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│                    ACTUAL FLOW (Broken)                             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Job Completes               post_result_to_watcher_thread()       │
│  ────────────  ─────────────▶  ┌─────────────────────────────┐    │
│  │ Blaxel    │               │ Format summary            │    │
│  │ Worker    │               │ + status emoji             │    │
│  └───────────┘               │ + result file (truncated) │    │
│                              │ + token/cost info         │    │
│                              └──────────┬──────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              message_handler.add_message()      │
│                              ┌─────────────────────────────┐    │
│                              │ Write to DB               │    │
│                              │ email_thread_messages     │    │
│                              └──────────┬──────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              ❌ MISSING: send_thread_update()     │
│                              ┌─────────────────────────────┐    │
│                              │ No WebSocket broadcast     │    │
│                              │ → No Redis pub/sub         │    │
│                              │ → Frontend never notified │    │
│                              └─────────────────────────────┘    │
│                                         │                        │
│                                         ▼                        │
│                              ┌─────────────────────────────┐    │
│                              │ User must refresh page     │    │
│                              │ → REST fetch loads messages │    │
│                              │ → Result finally appears   │    │
│                              └─────────────────────────────┘    │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

🐛 All Bugs Identified

Telegram Integration (7 bugs)

┌────────────────────────────────────────────────────────────────────┐
│  🔴 CRITICAL (Fix Immediately)                                     │
├────────────────────────────────────────────────────────────────────┤
│  1. Ghost Messages — Bot ignores ~50% of messages                 │
│     └─ User message appears → No response → "No messages here..."│
│                                                                     │
│  2. UI Stale State — Connect/disconnect doesn't refresh          │
│     └─ API operation succeeds → UI shows old data → Manual reload │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  🟡 HIGH (Before Customer Rollout)                                 │
├────────────────────────────────────────────────────────────────────┤
│  3. Silent Bot Replacement — New token replaces old without warning│
│     └─ connect() with new token → Old bot instantly disconnected │
│                                                                     │
│  4. Email Dependency — Bot needs Gmail for email tasks             │
│     └─ "Email access disconnected" error blocks functionality    │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  🟢 MEDIUM (Post-Launch)                                           │
├────────────────────────────────────────────────────────────────────┤
│  5. Orphaned Integrations — Each connect creates new record        │
│     └─ connect() → disconnect() → connect() = 3 DB records        │
│                                                                     │
│  6. Missing API Endpoints — /test, /send, /webhook return 404     │
│                                                                     │
│  7. API Prefix Inconsistency — Mixed /integrations/ vs /api/       │
└────────────────────────────────────────────────────────────────────┘

Workflow/Watcher (1 bug)

┌────────────────────────────────────────────────────────────────────┐
│  🔴 HIGH (Core Feature Broken)                                     │
├────────────────────────────────────────────────────────────────────┤
│  8. WebSocket Broadcast Missing — Results don't appear in real-time│
│     └─ add_message() writes to DB → No WebSocket → No UI update  │
└────────────────────────────────────────────────────────────────────┘

🎯 Current Plan & Priorities

Phase 1: Critical Bugs (Week 1)

┌────────────────────────────────────────────────────────────────────┐
│  Priority 1: Fix Ghost Message Bug                                 │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Investigate message processing pipeline                       │ │
│  │ → Check for race conditions / timeouts                       │ │
│  │ → Add logging for message deduplication                     │ │
│  │ → Test with different message types                          │ │
│  └──────────────────────────────────────────────────────────────┘ │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Expected Impact:                                             │ │
│  │ - Bot becomes reliable (100% response rate)                 │ │
│  │ - User trust restored                                        │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 2: Fix UI Stale State                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Add WebSocket/Reactive updates after connect/disconnect      │ │
│  │ → Implement polling fallback if WS unavailable               │ │
│  │ → Add visual loading indicators                              │ │
│  └──────────────────────────────────────────────────────────────┘ │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Expected Impact:                                             │ │
│  │ - Users see current integration state immediately            │ │
│  │ - Reduces confusion after config changes                    │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

Phase 2: High Priority Bugs (Week 2)

┌────────────────────────────────────────────────────────────────────┐
│  Priority 3: Add Bot Replacement Warning                            │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Check for existing bot before connecting new one             │ │
│  │ → Return 409 Conflict if bot exists                          │ │
│  │ → Or require ?force=true parameter                          │ │
│  │ → Add confirmation modal in UI                               │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 4: Fix WebSocket Broadcast (Watcher Notification)        │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Add send_thread_update() to add_message()                   │ │
│  │ → Mirror modify_message() behavior                         │ │
│  │ → Test with multi-instance Redis pub/sub                    │ │
│  │ → ~5 lines of code                                          │ │
│  └──────────────────────────────────────────────────────────────┘ │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Expected Impact:                                             │ │
│  │ - Watcher threads update in real-time                       │ │
│  │ - Workflow results appear immediately                        │ │
│  │ - Autopilot experience much smoother                        │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 5: Improve Email Dependency Error Handling              │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Make bot functional for non-email tasks without Gmail       │ │
│  │ → Clearer error messaging                                   │ │
│  │ → Suggest connecting Gmail when needed                       │ │
│  │ → Show which features require email vs. don't                │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

Phase 3: Medium Priority & UX (Week 3-4)

┌────────────────────────────────────────────────────────────────────┐
│  Priority 6: Fix Disconnect Cleanup                                │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Properly deactivate old integrations                         │ │
│  │ → Don't create duplicate records                             │ │
│  │ → Clean up webhooks                                          │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 7: Simplify Setup Flow                                   │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Add "Connect Telegram" button with token input               │ │
│  │ → Step-by-step BotFather instructions                       │ │
│  │ → Auto-verify webhook after connect                          │ │
│  │ → Success confirmation with bot details                      │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 8: Standardize API Prefixes                              │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Choose one pattern: /integrations/{service}/                │ │
│  │ → Migrate all endpoints to consistent pattern                │ │
│  │ → Update documentation                                       │ │
│  │ → Fix Gmail endpoint (currently 404 on all paths)           │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────┐
│  Priority 9: Add Skill Suggestions                                 │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │ Preload/suggest skills during autopilot setup               │ │
│  │ → Categorize skills (Email, Research, Content, etc.)        │ │
│  │ → Show available skills based on connected integrations      │ │
│  │ → One-click enable/disable                                  │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

🔧 Fix Options (WebSocket Broadcast Bug)

Option 1: Minimal Fix (Recommended) ✅

Location: core/agent/handlers/message_handler.py

async def add_message(
    thread_id: str,
    content: str,
    role: str,
    files: Optional[List[str]] = None,
    metadata: Optional[dict] = None,
) -> EmailThreadMessage:
    # ... existing database write code ...

    # ✅ ADD THIS LINE:
    await websocket_manager.send_thread_update(thread_id)

    return message

Pros:

  • Simple (~5 lines of code)
  • Mirrors existing modify_message() behavior
  • Fixes the immediate issue
  • Low risk

Cons:

  • Broadcasts ALL message additions (may be unnecessary for some use cases)
  • No filtering based on message type/role

Effort: 2-4 hours (including testing)


Option 2: Conditional Broadcast

async def add_message(..., should_broadcast: bool = True):
    # ... existing code ...

    if should_broadcast:
        await websocket_manager.send_thread_update(thread_id)

    return message

Pros:

  • More flexibility for callers
  • Can skip broadcast for internal/system messages

Cons:

  • More complex API surface
  • Requires updating all callers
  • Easy to forget to set should_broadcast=True

Effort: 4-6 hours (including updating callers)


Option 3: Broadcast-Only for System Messages

async def add_message(...):
    # ... existing code ...

    if role == "system":
        await websocket_manager.send_thread_update(thread_id)

    return message

Pros:

  • Only broadcasts system messages (workflow results, etc.)
  • Skips broadcast for user messages (handled elsewhere)

Cons:

  • Watcher's assistant response uses role="assistant" — wouldn't broadcast
  • Doesn't fix the Watcher auto-response issue
  • Harder to reason about

Effort: 3-5 hours (but doesn't fully fix the bug)


Option 4: Unified Message Handler

async def add_message(...):
    message = await db_write(...)

    # Unified broadcast logic
    await websocket_manager.send_thread_update(thread_id)

    return message

And remove duplicate broadcast from modify_message():

async def modify_message(...):
    message = await db_update(...)
    # ❌ REMOVE: await websocket_manager.send_thread_update(thread_id)
    # (Now handled in add_message/modify_message unified path)
    return message

Pros:

  • Cleanest architecture
  • No duplication
  • Broadcasts all updates consistently

Cons:

  • Requires refactoring modify_message() callers
  • Risk of breaking existing code
  • May introduce bugs during refactoring

Effort: 8-12 hours (high risk)


🏆 Recommended Comprehensive Fix

Primary Fix (Option 1) + Safeguards

Implementation:

# core/agent/handlers/message_handler.py

async def add_message(
    thread_id: str,
    content: str,
    role: str,
    files: Optional[List[str]] = None,
    metadata: Optional[dict] = None,
) -> EmailThreadMessage:
    """
    Add a message to a thread and broadcast WebSocket update.

    Args:
        thread_id: The thread ID to add the message to
        content: The message content
        role: Message role (user, assistant, system, etc.)
        files: Optional list of file IDs
        metadata: Optional metadata dictionary

    Returns:
        The created EmailThreadMessage
    """
    # Write to database
    message = await db.create_email_thread_message(
        thread_id=thread_id,
        content=content,
        role=role,
        files=files or [],
        metadata=metadata or {}
    )

    # Broadcast WebSocket update to all connected clients
    # This mirrors the behavior in modify_message() and modify_message_by_id()
    try:
        await websocket_manager.send_thread_update(thread_id)
    except Exception as e:
        # Log but don't fail - message is already in DB
        logger.error(f"Failed to broadcast thread update: {e}")

    return message

Why This Approach:

  1. Simple & Low Risk — Minimal code change, follows existing pattern
  2. Consistent — Mirrors modify_message() behavior exactly
  3. Robust — Error handling ensures broadcast failures don't break message creation
  4. Well-Documented — Clear docstring explaining the behavior
  5. Testable — Easy to verify broadcast happens

Complementary Changes:

# Add unit test for WebSocket broadcast
async def test_add_message_broadcasts_ws():
    # Create thread
    thread = await create_thread()

    # Mock WebSocket manager
    with patch('websocket_manager.send_thread_update') as mock_broadcast:
        # Add message
        msg = await add_message(thread_id=thread.id, content="test", role="user")

        # Verify broadcast was called
        mock_broadcast.assert_called_once_with(thread.id)

Rollback Plan:

If broadcast causes issues (e.g., performance, spam):

  • Feature flag the broadcast: if config.BROADCAST_MESSAGES: await ...
  • Or add rate limiting: if not rate_limiter.is_limited(thread_id): await ...

📊 Impact Analysis

Before Fix

User Experience:
├─ Open watcher thread
├─ Trigger workflow
├─ Wait 10-60 seconds
├─ Nothing happens...
├─ Refresh page
└─ Result appears 😤

Technical Impact:
├─ Messages written to DB ✅
├─ WebSocket never fires ❌
├─ Frontend never updates ❌
└─ Users think system is broken 😞

After Fix

User Experience:
├─ Open watcher thread
├─ Trigger workflow
├─ Wait 10-60 seconds
├─ ✨ Result appears automatically! ✨
└─ Happy user 😊

Technical Impact:
├─ Messages written to DB ✅
├─ WebSocket fires ✅
├─ Frontend updates in real-time ✅
└─ Users see system working as intended 🎉

Performance Considerations

Metric Before After Impact
DB writes 1 per message 1 per message No change
WebSocket broadcasts 0 per message 1 per message +1 Redis pub/sub
Client updates On refresh only Real-time Better UX
Network traffic No WS traffic ~1KB/message Negligible

Conclusion: Minimal performance impact, significant UX improvement.


🧪 Testing Strategy

Unit Tests

# Test 1: Broadcast happens on add_message
assert mock_send_thread_update.called_with(thread_id)

# Test 2: Broadcast doesn't fail if WS manager errors
# (should log error but still return message)

# Test 3: Multiple messages all broadcast
for i in range(5):
    await add_message(...)
assert mock_send_thread_update.call_count == 5

Integration Tests

# Test 1: Real-time update in browser
# - Open thread in browser (WS connected)
# - Trigger workflow via API
# - Assert message appears without refresh

# Test 2: Multiple tabs update
# - Open same thread in 3 tabs
# - Add message via API
# - Assert all 3 tabs update

# Test 3: Multi-instance Redis pub/sub
# - Backend instance A writes message
# - Backend instance B's client should receive broadcast

Manual QA Checklist

  • Open watcher thread in browser
  • Trigger workflow owned by watcher
  • Confirm message appears in real-time (no refresh)
  • Check browser console for WebSocket errors
  • Test with multiple browser tabs
  • Test Watcher's auto-response also broadcasts
  • Verify no performance degradation
  • Check Redis pub/sub logs

📋 Action Items

Immediate (This Week)

  • Engineering: Implement Option 1 fix in message_handler.py
  • Engineering: Add unit tests for WebSocket broadcast
  • QA: Set up test environment with Redis
  • QA: Execute integration tests
  • Engineering: Code review

Short Term (Next 2 Weeks)

  • Engineering: Deploy fix to staging
  • QA: Run full regression test suite
  • Engineering: Fix ghost message bug (Telegram)
  • Engineering: Fix UI stale state bug
  • Engineering: Add bot replacement warning

Medium Term (Month 1)

  • Engineering: Standardize API prefixes
  • Product: Simplify Telegram setup flow
  • Engineering: Implement skill suggestions
  • QA: Full end-to-end testing of all fixes

📚 Resources

Files

File Purpose Lines of Interest
core/agent/handlers/message_handler.py Contains bug add_message() function
jobs/tasks.py Calls buggy function 387-640
core/automation/routes.py Manual trigger endpoint 361-395
core/watcher/routes.py Watcher run-now endpoint 707-746
website/src/providers/threads-provider.tsx Frontend WS listener —
models.py Database schemas watcher_agents (1361), automated_jobs (847), email_threads (797)

Documentation

  • Bug Report: ~/vic-workspace/actrun-qa/bug-report-watcher-notification.md
  • Telegram Audit: ~/vic-workspace/actrun-qa/telegram-audit-2026-04-23.md
  • Original Investigation: ~/vic-workspace/actrun-qa/investigation-2026-04-16.md

Test Artifacts

  • Telegram bot token: /tmp/actrun-bot-token.txt
  • Test scripts: /tmp/ar-tg-*.mjs and /tmp/ar-workflow-*.mjs

🔗 Related Issues

  • Telegram Ghost Messages — Similar pattern: message created but no response
  • UI Stale State — Different manifestation of same root cause (missing updates)
  • Watcher Auto-Response — Also uses add_message(), would be fixed by same change

Created by: Vic (AI chief of staff) Date: April 23, 2026 Status: Ready for Engineering Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment