Designing a Notification System: A Practical Approach

You add a “someone liked your post” notification to your app. One event, one recipient, a simple database insert. That works fine for hundreds of users.

Then your app grows. A popular post gets five thousand likes in an hour. Each like should notify the post author — through whatever channels they have enabled. Your synchronous notification call, running inside the request handler, now blocks for seconds while it fires emails and push messages. Errors in the notification code start failing the like request itself.

That’s the fan-out problem: one event needs to trigger notifications for many recipients across multiple channels. Getting this right requires separating concerns that are easy to tangle — the event trigger, routing logic, template rendering, delivery, and user preferences.

I’m going to walk through each layer and show where the non-obvious decisions sit. By the end, you’ll have a clear mental model for building or evaluating a notification system that scales.

Series: Part 2 of 4. Start with Part 1: System Design Basics: Scaling from One User to One Million

On this page

• Four channels, four different contracts
• The notification service as a routing layer
• The fan-out problem
• Templates: render late, not early
• Delivery guarantees and deduplication
• User preferences: three layers
• What actually fails in production

Four channels, four different contracts

In-app, email, push, and SMS don’t just differ in how they reach the user — they have different reliability models, cost profiles, and user expectations.

In-app notifications are stored in your database and surfaced when the user opens the app. You can query for unread items and render a bell icon count or a notification feed. They’re only visible when someone is actively using the product, which means they accumulate. Your in-app notification table will need pagination and a way to mark items as read.

Email goes through your SMTP provider or a transactional email service. Delivery is generally reliable for valid addresses. It’s asynchronous — users see it when they check their inbox, which might be hours later. Email is the right channel for digests, order confirmations, and anything non-urgent. It’s also the hardest to unsend once the provider has accepted it.

Push notifications route through platform gateways: Apple Push Notification Service (APNs) for iOS and Firebase Cloud Messaging (FCM) for Android. The platform handles delivery to the device. The guarantee is best-effort — a device that’s offline for too long may never receive the notification. I covered the full push delivery path in Push Notifications in React Native: The Architecture Explained.

SMS routes through carrier networks and has the highest cost per message. It also has the strictest compliance requirements in several countries — unsubscribe mechanisms are legally required in many jurisdictions. Reserve SMS for high-urgency transactional messages.

Channel	Reliability	Cost	Best for
In-app	High (stored)	Minimal	Social activity, alerts
Email	High (for valid addresses)	Low	Digests, receipts, confirmations
Push	Best-effort (platform-dependent)	Low	Urgent, real-time alerts
SMS	High (when device reachable)	High	Critical transactional

The notification service as a routing layer

The mistake most teams make early on: each feature calls the email API or push API directly. The order confirmation service sends email. The comment service sends push. This works until you need to change providers, add a new channel, or apply user preferences consistently.

A notification service sits between features and delivery channels. It accepts notification events and routes them through per-channel queues.

The flow:

1. A feature emits a notification event to the notification service
2. The notification service checks user preferences for each channel
3. For each enabled channel, it enqueues a per-channel job
4. Workers process channel jobs independently: email worker, push worker, SMS worker, in-app writer

interface NotificationJob {
  id: string;
  userId: string;
  channel: "in-app" | "email" | "push" | "sms";
  templateKey: string;
  variables: Record<string, string>;
  idempotencyKey: string;
  scheduledFor?: Date;
}

The variables field holds the raw event data — not the rendered message. The templateKey points to a separate template. Why this matters is covered in the template section below.

Because each channel is an independent queue with its own worker, a push delivery failure doesn’t hold up the email, and an SMS backlog doesn’t block in-app writes. The Background Jobs and Message Queues post covers the queue semantics this system relies on, and BullMQ and Redis for Reliable Jobs shows a concrete implementation.

The fan-out problem

Here’s where the architecture earns its complexity.

Imagine a social feature: a user posts something and their ten thousand followers each need to be notified. Your event handler receives one “new post” event. You need to produce potentially ten thousand notification jobs across multiple channels. That’s fan-out.

The naive approach: inside the request handler, query all followers, loop through them, enqueue a job per user. For ten users, this takes milliseconds. For ten thousand, you’re holding the HTTP connection open while writing ten thousand queue entries. A partial failure midway leaves some followers notified and others not, with no clear recovery path.

The correct approach is a two-stage fan-out:

Stage 1 — the request handler enqueues a single “fan-out job” containing the event data and a recipient query.

Stage 2 — a fan-out worker runs the query, paginates through recipients, and enqueues per-user notification jobs in batches.

// Stage 1: the request handler enqueues one fan-out job
interface FanOutJob {
  eventType: string;
  sourceId: string;
  recipientQuery: {
    type: "followers";
    sourceUserId: string;
  };
  channels: ("push" | "email" | "in-app")[];
  templateKey: string;
  variables: Record<string, string>;
}

// Stage 2: the fan-out worker expands to per-user jobs
async function expandFanOut(
  job: FanOutJob,
  queue: Queue
): Promise<void> {
  const recipients = queryFollowers(job.recipientQuery.sourceUserId);
  for await (const batch of paginate(recipients, 500)) {
    const jobs = batch.map((userId) => buildNotificationJob(userId, job));
    await queue.addBulk(jobs);
  }
}

The request handler returns immediately. The fan-out job runs asynchronously. Each per-user job then runs its own delivery independently.

Before you continue: If a user has one million followers and each generates three channel jobs (push, email, in-app), how many queue entries does one fan-out event produce?

Three million. At one post per second from a popular account, queue depth matters — monitor lag, not just error rate.

For extremely large audiences, some teams skip write-time fan-out for the heaviest accounts and switch to read-time fan-out: storing one event record and letting clients query on load. That removes the queue pressure entirely but means you can’t push a notification to a device.

Check this before moving on

• Your request handler enqueues one fan-out job and returns immediately
• The fan-out worker paginates through recipients — no single query for all of them at once
• Per-user notification jobs are created by the fan-out worker, not by the request handler

Templates: render late, not early

When you enqueue a notification, don’t render the final message string immediately. Store the template key and the raw variables instead.

Store this	Not this
{ templateKey: "post_liked", variables: { likerName: "Alex" } }	"Alex liked your post"

Rendering at send time — late rendering — gives you several things:

• You can fix a typo in a template retroactively. All pending jobs pick up the change because they haven’t rendered yet.
• You can localize the message for each user’s preferred language at delivery time.
• You can A/B test notification copy without re-queuing events.
• The queue entry is smaller and language-agnostic.

If you render early and store the string, you’ve committed to that wording at enqueue time. A typo deployed at 9am affects every notification enqueued between then and the hotfix, with no way to correct the messages already sitting in the queue.

function renderNotification(
  template: NotificationTemplate,
  variables: Record<string, string>,
  locale: string
): string {
  const localizedBody =
    template.translations[locale] ?? template.default;
  return localizedBody.replace(
    /\{\{(\w+)\}\}/g,
    (_, key) => variables[key] ?? ""
  );
}

The template itself lives in a configuration store — a database table or file-based config — not in the queue payload. When the worker picks up a job, it loads the current template and renders using the stored variables. This is the design decision I’ve seen teams get wrong most often, because the “render early” path looks simpler until you need to iterate on copy.

Delivery guarantees and deduplication

Notification workers run with at-least-once delivery semantics. A worker might process the same job twice if it crashes after sending but before acknowledging the queue. A user who receives the same “Alex liked your post” twice will notice — and at scale, this happens constantly without a deduplication layer.

The fix is an idempotency key. Before sending, check whether a notification with the same key has already been delivered. Use a hash of the notification type, event ID, user ID, and channel.

import { createHash } from "node:crypto";

function makeIdempotencyKey(
  notificationType: string,
  eventId: string,
  userId: string,
  channel: string
): string {
  return createHash("sha256")
    .update([notificationType, eventId, userId, channel].join(":"))
    .digest("hex");
}

async function shouldSendNotification(
  redis: RedisClient,
  key: string,
  ttlSeconds = 86_400
): Promise<boolean> {
  // NX = only set if key does not already exist
  const result = await redis.set(key, "1", "NX", "EX", ttlSeconds);
  return result === "OK";
}

If shouldSendNotification returns false, the job was already processed — skip the send and acknowledge the queue. This makes workers idempotent without a separate deduplication database.

The TTL limits Redis memory footprint. Twenty-four hours is usually sufficient: a retry arriving 25 hours late is a different problem worth handling separately. For jobs that fail permanently — bad phone numbers, bounced email addresses — route them to a dead-letter queue and handle those failures through a monitoring process rather than retrying indefinitely.

User preferences: three layers

Users don’t want to “turn off notifications.” They want to control which types arrive on which channels. Someone who turns off email marketing doesn’t want to lose password reset emails. Someone who disables push for quiet hours still wants in-app badges to work.

Three layers to model:

Channel-level — the user has enabled or disabled the entire channel. Push disabled means no push, regardless of event type.

Category-level — per notification type, per channel. “Marketing emails: off. Transactional emails: on. Push for social activity: off. Push for direct messages: on.”

Quiet hours — push delivery is suppressed between specific hours in the user’s timezone. The notification job is still created and stored in-app; only the push delivery is delayed or skipped.

interface UserPreferences {
  userId: string;
  channels: {
    push: boolean;
    email: boolean;
    sms: boolean;
    inApp: boolean;
  };
  categories: Record<string, {
    push: boolean;
    email: boolean;
    sms: boolean;
  }>;
  quietHours?: {
    start: number;   // hour 0–23 in user's timezone
    end: number;
    timezone: string;
  };
}

function canSend(
  prefs: UserPreferences,
  channel: keyof UserPreferences["channels"],
  category: string
): boolean {
  if (!prefs.channels[channel]) return false;

  const catPrefs = prefs.categories[category];
  if (!catPrefs) return true; // default allow when no preference set

  return catPrefs[channel as keyof typeof catPrefs] ?? true;
}

Check preferences before enqueuing per-channel jobs — not inside the channel worker. Filtering at the notification service level keeps the worker queues lean and avoids wasted work.

Preference state	Push job created?	Email job created?
All push disabled	No	Yes (if email enabled)
Marketing category off, email only	Yes	No
Quiet hours active	No (delayed or skipped)	Yes

One edge case worth calling out: transactional notifications — receipts, password resets, account security alerts — should bypass category preferences but still respect channel-level opt-outs. A user who disables all email should not receive a receipt email, but a user who’s turned off marketing email should still get their order confirmation.

What actually fails in production

The fan-out worker is the most common failure point, not the individual channel workers. If a popular account’s fan-out job produces half a million jobs per minute and your workers can process two hundred thousand, the backlog grows and notifications arrive with meaningful delay. Monitor queue depth and job age, not just error rates.

The second failure: template rendering errors from missing variables. A notification job enqueued without the required variables renders as empty or garbled text. Validate that all required variables are present when the notification event is created — not at render time inside the worker.

The third: stale push tokens. When a user reinstalls the app, they get a new push token. The old token produces a BadDeviceToken error from APNs or a similar error from FCM. Your push worker needs to handle this response by removing the stale token from the user’s record rather than retrying the same invalid token on the next job.

Preferences stored as a single JSON column against the user record make preference lookups fast but make bulk updates slow — if you need to toggle a category for all users after a product change, you’re updating every row. A normalized preferences table is slower to read but easier to update in bulk. Neither is wrong; both trade-offs have their place depending on your read-to-write ratio.

The notification layer is a product, not just infrastructure. Each notification is a bid for the user’s attention. Send too many and users revoke the channel entirely — push permission, once denied on iOS, requires the user to go into Settings to restore. Send too few and users miss things that matter to them.

The architecture above — separated channels, per-channel queues, late-rendered templates, idempotency, layered preferences — exists to serve that goal: the right message on the right channel at the right time, delivered exactly once.

Sources

• APNs overview and authentication — Apple Developer Documentation
• Firebase Cloud Messaging architecture — Firebase Documentation
• Fan-out on write vs fan-out on read — Martin Kleppmann, Designing Data-Intensive Applications
• Idempotency and at-least-once delivery — exactly-once semantics in distributed systems
• TCPA compliance for SMS notifications — Federal Communications Commission