- Overview
- Goals & Non-Goals
- Reliability Guarantees
- Schema Design
- System Architecture
- End-to-End Flow
- Worker Architecture
- Circuit Breaker Design
- Provider Abstraction
- Failure Modes & Graceful Degradation
- Open Questions
This document describes the architecture for a multi-channel notification system capable of reliably delivering push, SMS, and email notifications to 1M+ users.
The system is designed around three core concerns:
- No missed sends — every triggered notification must be durably recorded and eventually delivered
- No duplicate sends — deduplication is enforced at the schema level and at the provider level via idempotency keys
- Graceful degradation — provider failures are detected automatically, and the system routes around them without manual intervention
- Deliver notifications across push, SMS, and email channels
- Tolerate provider outages without data loss or duplicate delivery
- Support 1M+ users with horizontally scalable workers
- Provide observability into delivery status, failure rates, and provider health
- Allow new providers to be onboarded without changes to core delivery logic
- Real-time delivery (sub-second SLA) — this system targets eventual delivery with high reliability, not low latency
- In-app notifications (separate concern, no external provider dependency)
- User notification preferences / suppression rules (can be layered on top)
- Billing or rate-limit enforcement per user
| Guarantee | Mechanism |
|---|---|
| No missed sends | Transactional Outbox Pattern — notification + outbox row written in same DB transaction |
| No duplicate sends | unique(hash, user_id) on notifications; idempotency keys on provider requests |
| Ambiguous send resolution | Reconciliation worker (TSQ pattern) — queries provider for ground truth before retrying |
| Provider failure isolation | Circuit breaker per (channel, provider) — failures in one do not affect others |
| Worker crash recovery | SELECT FOR UPDATE SKIP LOCKED — unclaimed or expired jobs are automatically reclaimable |
Represents the intent to notify a user. Created once per triggering event.
CREATE TABLE notifications (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL,
title VARCHAR(100),
message VARCHAR(255),
html_content TEXT,
hash VARCHAR(64) NOT NULL, -- SHA-256 of (user_id + channel + core payload)
status VARCHAR(50) NOT NULL DEFAULT 'pending',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uq_notification_hash UNIQUE (hash, user_id)
);
CREATE INDEX idx_notifications_user_id ON notifications(user_id);
CREATE INDEX idx_notifications_status ON notifications(status);Notes:
hashis a SHA-256 of the notification's user-scoped payload. TheUNIQUE(hash, user_id)constraint makes duplicate notification creation a no-op at the DB level — no application-layer guard needed.titleserves double duty: email subject line and push notification title.html_contentis nullable — SMS and push notifications do not use it.statusvalues:pending,processing,delivered,failed- Enum types are intentionally avoided in the database in favour of
VARCHARwith application-layer validation. This keeps the schema flexible and avoids costlyALTER TYPEmigrations.
Tracks each delivery attempt per channel per notification. A single notification may produce multiple attempts (e.g. one for email, one for SMS), and each attempt may be retried.
Recipient data is denormalized at creation time — see Section 4.4 for rationale.
CREATE TABLE notification_attempts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
notification_id UUID NOT NULL REFERENCES notifications(id),
channel VARCHAR(50) NOT NULL, -- 'email' | 'sms' | 'push'
provider VARCHAR(50), -- 'sendgrid' | 'twilio' | 'firebase' etc.
last_used_provider VARCHAR(50), -- tracks last attempted provider for circuit breaker awareness
recipient VARCHAR(255) NOT NULL, -- email address, phone number, device token, or webhook URL
idempotency_key UUID NOT NULL DEFAULT gen_random_uuid(), -- sent to provider to prevent duplicate delivery
provider_message_id VARCHAR(255), -- ID returned by provider on successful submission
status VARCHAR(50) NOT NULL DEFAULT 'pending',
attempts INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 3,
locked_at TIMESTAMPTZ, -- set when a worker claims this attempt
failed_at TIMESTAMPTZ,
successful_at TIMESTAMPTZ,
reconciled_at TIMESTAMPTZ, -- last time the reconciliation worker verified status with provider
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_attempts_notification_id ON notification_attempts(notification_id);
CREATE INDEX idx_attempts_status ON notification_attempts(status);
CREATE INDEX idx_attempts_channel ON notification_attempts(channel);Notes:
idempotency_keyis generated at row creation and sent with every request to the provider. If a worker retries after a transient failure, the same key is reused — a well-behaved provider will deduplicate on their end.provider_message_idenables the reconciliation worker (TSQ pattern) to query the provider for ground truth on ambiguous attempts.last_used_providertells the retry logic which provider was last tried, preventing a retry from hitting an already-tripped circuit.statusvalues:pending,processing,delivered,failed,reconciling
Written in the same database transaction as notifications. A relay worker reads from this table and publishes to the message broker. This decouples durability (DB transaction) from broker availability.
CREATE TABLE outbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
notification_id UUID NOT NULL REFERENCES notifications(id),
payload JSONB NOT NULL,
published_at TIMESTAMPTZ, -- NULL until relay worker successfully publishes
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_outbox_published_at ON outbox(published_at) WHERE published_at IS NULL;Notes:
- The partial index on
published_at WHERE NULLmeans the relay worker's poll query only scans unpublished rows — the index shrinks as rows are processed. published_atalso doubles as a relay lag monitor. A rising count of old unpublished rows is an alerting signal.
At 1M+ users, looking up recipient data (email, phone number, device tokens) from a users table at delivery time creates two problems:
- Contention — delivery workers and the core user service compete for reads on the same hot tables under load.
- Coupling — a users table migration, schema change, or outage now also breaks the notification pipeline.
By writing recipient onto notification_attempts at creation time, workers become self-contained. They need only the attempts table to execute a delivery. An added benefit: the recipient is snapshotted as it was at notification creation time, which is the correct semantic — a mid-flight email address change should not silently redirect a notification already in flight.
┌─────────────────────────────────────────────┐
│ Application Layer │
│ (API / Event / Scheduler triggers notify) │
└──────────────────┬──────────────────────────┘
│
┌──────────────▼──────────────┐
│ PostgreSQL │
│ │
│ ┌─────────────────────┐ │
│ │ notifications │ │
│ └─────────────────────┘ │
│ ┌─────────────────────┐ │ ← single transaction
│ │ outbox │ │
│ └─────────────────────┘ │
│ ┌─────────────────────┐ │
│ │ notification_attempts│ │
│ └─────────────────────┘ │
└──────────────┬──────────────┘
│
┌───────────────────────▼──────────────────────────┐
│ Outbox Relay Worker │
│ SELECT FOR UPDATE SKIP LOCKED on outbox │
│ Publishes to message broker, marks published_at │
└───────────────────────┬──────────────────────────┘
│
┌──────────────▼──────────────┐
│ Message Broker │
│ (SQS / RabbitMQ) │
└──────────────┬──────────────┘
│
┌───────────────────────▼──────────────────────────┐
│ Delivery Worker Pool │
│ SELECT FOR UPDATE SKIP LOCKED on attempts │
│ Checks circuit breaker state in Redis │
│ Dispatches via Provider Factory │
└──────┬─────────────────────────┬─────────────────┘
│ │
┌────────────────▼────┐ ┌────────▼──────────────────┐
│ Provider Factory │ │ Circuit Breaker (Redis) │
│ email / sms / push │ │ keyed by (channel, prov) │
└──────┬──────┬───────┘ └───────────────────────────┘
│ │
┌──────────▼──┐ ┌─▼──────────┐ ┌──────────────┐
│ SendGrid │ │ Twilio │ │ Firebase │
│ Mailgun │ │ Vonage │ │ APNs │
└─────────────┘ └────────────┘ └──────────────┘
│
┌───────────────────────▼──────────────────────────┐
│ Reconciliation Worker (TSQ) │
│ Polls ambiguous attempts (no provider_message_id)│
│ Queries provider status API for ground truth │
│ Marks delivered / resets to pending for retry │
└──────────────────────────────────────────────────┘
1. Application triggers notification
│
▼
2. Open DB transaction:
a. INSERT INTO notifications (with hash)
→ if UNIQUE(hash, user_id) conflicts: silently abort (duplicate)
b. INSERT INTO notification_attempts (one row per channel, with denormalized recipient + idempotency_key)
c. INSERT INTO outbox (payload referencing notification_id)
COMMIT
│
▼
3. Outbox Relay Worker polls outbox WHERE published_at IS NULL
→ SELECT FOR UPDATE SKIP LOCKED (batch of N rows)
→ Publishes each to message broker
→ UPDATE outbox SET published_at = NOW()
│
▼
4. Delivery Worker picks up job from broker
→ SELECT FOR UPDATE SKIP LOCKED on notification_attempts WHERE status = 'pending'
→ Check circuit breaker in Redis for (channel, provider)
│
├── Circuit OPEN → skip provider, try fallback provider
│ if no fallback available → requeue with backoff
│
└── Circuit CLOSED / HALF-OPEN →
Send via provider (include idempotency_key in request)
│
├── Success →
│ UPDATE attempts SET status='delivered',
│ provider_message_id=<id>,
│ successful_at=NOW()
│ Record success in circuit breaker state
│
└── Failure →
If transient (timeout, 5xx):
UPDATE attempts SET attempts=attempts+1, last_used_provider=<p>
If attempts < max_attempts → reset to pending (reclaimable)
If attempts >= max_attempts → SET status='failed'
Record failure in circuit breaker state
If failure threshold crossed → OPEN circuit for (channel, provider)
│
▼
5. Reconciliation Worker (TSQ) runs periodically
→ Queries attempts WHERE status='processing'
AND provider_message_id IS NULL
AND updated_at < NOW() - INTERVAL '2 minutes'
→ Calls provider status API with idempotency_key or provider_message_id
→ If confirmed delivered → mark 'delivered'
→ If confirmed failed → reset to 'pending' or mark 'failed'
→ UPDATE reconciled_at = NOW()
Responsibility: Bridge the database and the message broker. This is the only component that reads from the outbox table.
Behaviour:
- Polls
outbox WHERE published_at IS NULLin batches - Uses
SELECT FOR UPDATE SKIP LOCKEDto safely claim rows across multiple relay instances - On broker unavailability: backs off exponentially, does not crash — unprocessed rows remain in the outbox and are picked up on recovery
- On success: sets
published_at - Monitors relay lag (age of oldest unpublished row) as an alerting metric
Responsibility: Claim and execute notification delivery attempts.
Behaviour:
- Subscribes to message broker queue
- Claims batches of
notification_attempts WHERE status = 'pending'usingSELECT FOR UPDATE SKIP LOCKED - Pessimistic locking is appropriate here: the claiming operation is a simple point-in-time grab with no complex join, so deadlock risk is negligible
- Checks circuit breaker state in Redis before dispatching
- Delegates to provider via the Provider Factory
- Updates attempt status, records
provider_message_id, increments attempt counter - On failure below
max_attempts: resetsstatustopendingso the row is reclaimable - Horizontally scalable — add workers to increase throughput
Responsibility: Resolve ambiguous delivery attempts — cases where a message was likely sent but no confirmation was written (e.g. worker crashed post-send).
Behaviour:
- Runs on a schedule (e.g. every 2 minutes)
- Queries
notification_attempts WHERE status = 'processing' AND updated_at < NOW() - INTERVAL '2 minutes' - Calls provider status API using
idempotency_keyorprovider_message_id - Only transitions status after receiving a verifiable response — never assumes
- Updates
reconciled_aton each check - If provider status API is itself unavailable, leaves the row in
processingand retries on the next cycle
This pattern is borrowed from financial systems (Transaction Status Query) and ensures the system never makes an irreversible decision (retry or discard) based on an ambiguous state.
Circuit breaker state is stored in Redis, keyed per (channel, provider). This makes state shared across all delivery workers without DB overhead.
circuit:{channel}:{provider}
Examples:
circuit:sms:twilio
circuit:email:sendgrid
circuit:push:firebase
Each key holds a small JSON object:
{
"state": "closed", // closed | open | half-open
"failure_count": 0,
"last_failure_at": null,
"opened_at": null
} failures >= threshold
CLOSED ──────────────────────► OPEN
▲ │
│ │ cooldown period elapsed
│ ▼
└────────────────────── HALF-OPEN
probe succeeds
| State | Behaviour |
|---|---|
closed |
Normal operation. Failures are counted. |
open |
All requests for this (channel, provider) are rejected immediately. Fallback provider is used. |
half-open |
One probe request is allowed through. Success → closed. Failure → back to open. |
| Parameter | Value |
|---|---|
| Failure threshold to open | 5 consecutive failures |
| Cooldown before half-open | 60 seconds |
| Probe timeout | 10 seconds |
These should be configurable per provider, not hardcoded.
When a circuit is open, the delivery worker consults a fallback chain defined per channel:
email: sendgrid → mailgun → ses
sms: twilio → kudisms → bulksms
push: firebase → apns (platform-dependent)
If all providers in the chain are open, the attempt is requeued with exponential backoff and an alert is fired.
All providers implement a common interface. The delivery worker never calls a provider directly — it always goes through the factory.
interface NotificationProvider {
channel: Channel; // 'email' | 'sms' | 'push'
slug: string; // 'sendgrid' | 'twilio' | 'firebase' etc.
send(payload: NotificationPayload): Promise<ProviderResult>;
getStatus(messageId: string): Promise<ProviderStatus>; // used by reconciliation worker
}
interface NotificationPayload {
recipient: string;
title?: string;
message: string;
htmlContent?: string;
idempotencyKey: string;
}
interface ProviderResult {
success: boolean;
providerMessageId?: string;
error?: string;
}
type ProviderStatus = 'delivered' | 'failed' | 'pending' | 'unknown';class NotificationProviderFactory {
private providers: Map<string, NotificationProvider>;
getProvider(channel: Channel, providerSlug: string): NotificationProvider {
const key = `${channel}:${providerSlug}`;
const provider = this.providers.get(key);
if (!provider) throw new Error(`No provider registered for ${key}`);
return provider;
}
getAvailableProviders(channel: Channel): NotificationProvider[] {
// Returns the fallback chain for a channel, ordered by priority
// Filtered by circuit breaker state at call time
}
}Onboarding a new provider requires only implementing NotificationProvider and registering it — no changes to delivery worker logic.
| Failure | Impact | System Response |
|---|---|---|
| DB unavailable at notification creation | Notification not created | Transaction rolls back cleanly. Caller retries. No partial state. |
| Broker unavailable at relay time | Outbox rows accumulate | Relay worker backs off. Rows are safe in DB. Delivered in order on recovery. |
| Worker crash mid-send | Attempt stuck in processing |
Reconciliation worker resolves via provider status API after timeout. |
| Provider returns 5xx | Delivery fails transiently | Attempt reset to pending, failure counted in circuit breaker. Retry on next worker cycle. |
| Provider circuit opens | That (channel, provider) unavailable |
Fallback provider chain consulted. Workers skip open circuits without blocking. |
| All providers for a channel open | Channel temporarily undeliverable | Attempts requeued with backoff. Alert fired. No data lost. |
| Provider sends duplicate on retry | Duplicate delivery | Idempotency key on provider request prevents duplicate on well-behaved providers. |
| Reconciliation worker unavailable | Ambiguous attempts linger | Attempts remain in processing. No incorrect retries are triggered. Worker self-heals on restart. |
These items are deliberately deferred and should be resolved before implementation:
- Notification preferences — should users be able to opt out of specific channels? If so, where is that enforced — at creation time or at dispatch time?
- Retention policy — how long are
notification_attemptsrows kept? At 1M+ users with multiple channels, this table will grow fast. - Observability stack — what tooling is used for metrics and alerting? Circuit breaker state and relay lag should be first-class metrics.
max_attemptsper channel — should push, SMS, and email have different retry limits given their different cost and delivery semantics?