System Design: Payment Processing System

A capstone system design walkthrough — designing a payment processing system end to end — covering the core domain model, the ledger as the system's source of truth, idempotency and exactly-once-effect guarantees, integrating with external payment gateways and card networks, handling asynchronous webhooks, reconciliation, fraud and risk checks, and the specific correctness and compliance demands that make payments a uniquely unforgiving system design problem.

Table of Contents

1. Introduction
2. Why Payment Systems Are a Different Kind of Hard
3. The Core Domain Model
4. The Ledger: Double-Entry Bookkeeping as the Source of Truth
5. Idempotency: The Single Most Important Property
6. Integrating with Payment Gateways and Card Networks
7. The Payment State Machine
8. Webhooks: Handling Asynchronous Gateway Callbacks
9. The Saga: Coordinating Payment Across Multiple Services
10. Reconciliation
11. Fraud and Risk Checks
12. Data Security and Compliance
13. Consistency, Availability, and the CAP Trade-off for Money
14. Scaling the System
15. Observability for a Payment System
16. Common Pitfalls
17. Quick Reference Table
18. Conclusion

Introduction

A payment processing system takes the general system design vocabulary covered in this series' System Design guide — databases, caching, queues, load balancing — and applies it to a domain where the ordinary consequences of a bug are dramatically higher: a double-charged customer, a lost payment, or a corrupted ledger isn't a degraded user experience, it's real money moved incorrectly, sometimes irreversibly. This guide walks through designing such a system end to end, drawing directly on this series' DDD, Event-Driven Architecture, Database Migrations, and Secret Management guides, each of which turns out to be load-bearing infrastructure for getting payments right rather than optional architectural polish.

Client → Payment API → [validate, risk-check] → Payment Gateway (Stripe/Adyen/etc.) → Card Network → Bank
                              ↓                           ↓ (async webhook)
                          Ledger (source of truth)  ←  Payment State Machine

1. Why Payment Systems Are a Different Kind of Hard

The cost of a bug is measured in money, not just user experience

Most systems covered in this series can tolerate a transient bug with a bounded, recoverable cost — a stale cache entry, a brief outage, a duplicate email. A payment system's failure modes are different in kind: a duplicate charge is real money taken from a real customer without their consent; a lost payment confirmation can mean a customer paid but never received their order, or a merchant shipped goods without ever being paid. This is why idempotency (Section 4) and the ledger's correctness (Section 3) dominate this guide's concerns more than raw throughput does.

Money must reconcile — silently "close enough" isn't a valid state

A social media "likes" counter being off by one, briefly, is invisible and harmless.
A ledger being off by one cent, ANYWHERE, is a genuine defect that must be found and explained.

Unlike most eventually-consistent systems covered in this series' Event-Driven Architecture guide, where "briefly stale, then converges" is an acceptable trade-off, a payment ledger must reconcile to the cent, provably, against the external systems (card networks, banks) it represents — this is a stricter correctness bar than "eventually consistent," and Section 9's reconciliation process exists specifically to enforce it continuously, not just trust that it holds.

You are almost never processing the actual money movement yourself

A critical, freeing realization for the design that follows: a payment processing system, in the overwhelming majority of real-world designs, does not itself move money between bank accounts — it orchestrates a request to a payment gateway (Stripe, Adyen, Braintree, or a similar processor), which in turn talks to card networks (Visa, Mastercard) and banks. Your system's job is to reliably record intent, submit the request, track the outcome, and maintain an accurate internal ledger of what happened — not to reimplement banking infrastructure, which is precisely the kind of "don't build what a specialized provider already does well" guidance echoed in this series' Secret Management and OAuth2/OIDC guides for identity, applied here to money movement.

2. The Core Domain Model

Modeled with DDD, per this series' companion guide

public record PaymentId(Guid Value);
public record Money(long MinorUnits, string Currency); // e.g., 4999 minor units + "USD" = $49.99 — see Section 3's note on this

public enum PaymentStatus { Initiated, Authorized, Captured, Failed, Refunded, PartiallyRefunded }

public class Payment // the AGGREGATE ROOT, per this series' DDD guide
{
    public PaymentId Id { get; }
    public Money Amount { get; }
    public PaymentStatus Status { get; private set; }
    private readonly List<PaymentEvent> _domainEvents = new();

    public void Authorize(string gatewayAuthorizationId)
    {
        if (Status != PaymentStatus.Initiated)
            throw new InvalidOperationException($"Cannot authorize a payment in status {Status}");
        Status = PaymentStatus.Authorized;
        _domainEvents.Add(new PaymentAuthorizedEvent(Id, gatewayAuthorizationId));
    }

    public void Capture()
    {
        if (Status != PaymentStatus.Authorized)
            throw new InvalidOperationException($"Cannot capture a payment in status {Status}");
        Status = PaymentStatus.Captured;
        _domainEvents.Add(new PaymentCapturedEvent(Id, Amount));
    }
}

This directly applies this series' DDD guide's aggregate pattern — Payment is the aggregate root, enforcing its own state transitions (you cannot capture a payment that was never authorized) rather than trusting every caller to check status before mutating it, and raising domain events at exactly the points those transitions genuinely occur.

Why money should never be a floating-point or plain decimal type without care

// ❌ Floating-point arithmetic on money is a well-known, serious source of rounding errors
double amount = 49.99; // binary floating point cannot represent this exactly

// ✅ Store money as an integer count of the smallest currency unit (cents, minor units)
public record Money(long MinorUnits, string Currency); // 4999 minor units = $49.99

Representing money as double risks genuine, real rounding errors accumulating over many operations — the standard, widely-adopted practice is storing an amount as an integer number of the currency's smallest unit (cents for USD, pence for GBP), only converting to a display-formatted decimal string at the presentation layer, never performing arithmetic in that display format. decimal in C# is safer than double for money (base-10, not binary floating point), but many production payment systems still prefer integer minor units specifically for unambiguous cross-language, cross-system interoperability — worth being deliberate about which convention a given system adopts and applying it consistently everywhere money is represented.

Value objects for currency-safety

public Money Add(Money other)
{
    if (Currency != other.Currency) throw new InvalidOperationException("Cannot add different currencies");
    return this with { MinorUnits = MinorUnits + other.MinorUnits };
}

As covered in this series' DDD guide's value object discussion, wrapping a raw amount in a Money value object that enforces currency-matching on any arithmetic operation prevents an entire class of bugs (accidentally adding USD to EUR) at the type level, rather than relying on every call site to remember to check currencies match.

3. The Ledger: Double-Entry Bookkeeping as the Source of Truth

Why a simple "balance" column is insufficient

-- ❌ A single mutable balance column has no audit trail and is trivially corruptible by a single bad UPDATE
UPDATE accounts SET balance = balance - 100 WHERE id = 1;

A payment system needs more than "what is the current balance" — it needs an immutable, auditable record of every movement of money that ever occurred, and the ability to prove, at any point, exactly how the current balance was arrived at. A mutable balance column, updated in place, destroys that history the moment it's overwritten, and provides no structural protection against a bug (or a malicious actor) silently corrupting a balance with no trace of how it happened.

Double-entry bookkeeping: every movement recorded as two balanced entries

CREATE TABLE ledger_entries (
    id BIGINT PRIMARY KEY,
    transaction_id UUID NOT NULL,   -- groups the debit and credit belonging to one logical movement
    account_id BIGINT NOT NULL,
    amount_minor_units BIGINT NOT NULL, -- positive for a credit, negative for a debit
    currency CHAR(3) NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- A $49.99 payment captured: money moves from "customer owes" to "merchant receivable"
INSERT INTO ledger_entries (transaction_id, account_id, amount_minor_units, currency) VALUES
  ('a1b2c3d4-...', /* customer_receivable_account */ 100, -4999, 'USD'),
  ('a1b2c3d4-...', /* merchant_payable_account */    200,  4999, 'USD');
-- these two rows, sharing one transaction_id, must ALWAYS sum to zero

Double-entry bookkeeping — the centuries-old accounting technique this system borrows directly — records every movement of money as (at least) two balanced entries: a debit from one account and a credit to another, always summing to exactly zero for any given transaction. This isn't accounting ceremony for its own sake; it's a structural, mathematically-verifiable invariant: at any point, summing every ledger entry for a given transaction ID must equal zero, and summing every entry for a given account gives that account's genuine, provable current balance, derived entirely from the append-only history rather than trusted as a separately-maintained, corruptible number.

The ledger table is append-only, never updated or deleted

-- Correcting a mistake means inserting a NEW, compensating entry — never UPDATE or DELETE an existing row
INSERT INTO ledger_entries (transaction_id, account_id, amount_minor_units, currency) VALUES
  ('correction-e5f6...', /* customer_receivable_account */ 100, 4999, 'USD'),  -- reverses the original debit
  ('correction-e5f6...', /* merchant_payable_account */    200, -4999, 'USD'); -- reverses the original credit

This directly echoes the append-only log philosophy covered in this series' Kafka and Event Sourcing (via the DDD and Event-Driven Architecture guides) discussions — the ledger is never mutated in place; a mistake is corrected by inserting a new, compensating entry that reverses the original, preserving the complete, honest history of everything that happened, including the mistake and its correction, rather than erasing evidence that a mistake occurred at all. This property is what makes the ledger auditable and, critically, what regulators and auditors expect from any genuine financial system.

Balance as a derived, always-recomputable value

SELECT SUM(amount_minor_units) AS current_balance
FROM ledger_entries
WHERE account_id = 100;

An account's current balance is always a SUM query over its ledger entries — never a separately-stored, independently-updatable number that could drift out of sync with the entries that supposedly produced it. For performance (summing potentially millions of historical entries on every balance check is genuinely expensive), a cached/materialized balance is a reasonable optimization (directly connecting to this series' caching and materialized-view discussions), but it must always be treated as a derived cache of the ledger's truth, recomputable and re-verifiable against it at any time — never the authoritative source itself.

4. Idempotency: The Single Most Important Property

Why this is even more critical here than in any other system covered in this series

As covered throughout this series' RabbitMQ, Kafka, Azure Service Bus, and Event-Driven Architecture guides, every messaging technology provides at-least-once delivery, and every network call can time out ambiguously (did the request actually succeed server-side, or not, before the client gave up waiting?) — for most systems, a resulting duplicate action is an annoyance (a duplicate email, a slightly wasted computation). For a payment system, an un-idempotent retry means charging a customer twice for the same purchase, which is precisely why idempotency is this guide's single most emphasized property.

Idempotency keys: the standard mechanism

[HttpPost("/payments")]
public async Task<IActionResult> CreatePayment(
    [FromHeader(Name = "Idempotency-Key")] string idempotencyKey,
    CreatePaymentRequest request)
{
    var existing = await _idempotencyStore.GetResultAsync(idempotencyKey);
    if (existing is not null)
    {
        return Ok(existing); // the SAME response as the original request, no new charge attempted
    }

    var payment = await _paymentService.ProcessAsync(request);
    await _idempotencyStore.SaveResultAsync(idempotencyKey, payment);
    return Ok(payment);
}

This is the concrete implementation of the idempotency pattern introduced generally in this series' Redis guide's rate-limiting section and REST guide's discussion — a client generates a unique idempotency key for each logical payment attempt (not regenerated on retry) and includes it on every request, including retries; the server checks whether that key has already been processed and, if so, returns the original result rather than attempting the charge again. This is precisely how Stripe, Adyen, and every major payment gateway's own API is designed, and any payment system built on top of one should propagate this exact same discipline to its own client-facing API.

Idempotency at every layer the payment touches, not just the outermost API

Client → Payment API (idempotency key checked here)
            → Payment Gateway call (the GATEWAY also expects and enforces its own idempotency key)
            → Ledger write (a database-level unique constraint on transaction_id prevents a duplicate insert)
            → Event published (per this series' Event-Driven Architecture guide, consumers must ALSO be idempotent)

Idempotency needs to be enforced at every hop, not just the client-facing entry point — the call to the external payment gateway itself should include its own idempotency key (most major gateways support and expect this natively), the ledger write should have a database constraint preventing a duplicate transaction ID from ever being inserted twice, and any downstream event consumers (per this series' Event-Driven Architecture guide) must independently be idempotent against redelivery, since a payment system is exactly the kind of system where "we'll just be extra careful" is not an acceptable substitute for structural, enforced guarantees at every layer.

5. Integrating with Payment Gateways and Card Networks

The layers between your system and an actual bank

Your system → Payment Gateway (Stripe, Adyen, Braintree) → Card Network (Visa, Mastercard) → Issuing Bank

A payment gateway is the specialized third party that actually handles the sensitive complexity of talking to card networks and banks — authorization, settlement, PCI compliance for card data handling (Section 11) — so that a payment system, in the overwhelming majority of real designs, never directly touches raw card numbers or talks to a card network itself at all.

Authorization vs. capture: a two-phase pattern most gateways support

// Phase 1: authorize — places a hold on the customer's funds, doesn't yet move money
var authResult = await _gateway.AuthorizeAsync(new AuthorizeRequest(amount, cardToken));
payment.Authorize(authResult.GatewayAuthorizationId);

// Phase 2: capture — actually moves the money, typically once the order genuinely ships
var captureResult = await _gateway.CaptureAsync(authResult.GatewayAuthorizationId, amount);
payment.Capture();

Separating authorization (verifying funds are available and placing a hold) from capture (actually completing the charge) is a deliberate, widely-used design pattern — it lets a merchant confirm a customer can pay before committing to ship an order, and only finalize the charge once the order genuinely ships, reducing the need for refunds on orders that turn out to be unfulfillable, and directly mapping onto the Payment aggregate's state machine from Section 2.

Using gateway-provided tokens, never touching raw card numbers directly

// Card details are tokenized CLIENT-SIDE, by the gateway's own JS SDK — your server NEVER sees the raw card number
const { token } = await stripe.createToken(cardElement);
// only this opaque token is ever sent to YOUR backend

The standard, essentially universal pattern: raw card numbers are tokenized directly in the client (browser or mobile app), by the payment gateway's own SDK, before ever reaching your server — your backend only ever handles an opaque token representing the card, never the actual card number itself. This dramatically reduces your own system's PCI compliance burden (Section 11) since sensitive card data structurally never touches your infrastructure at all.

6. The Payment State Machine

An explicit, enumerable set of states and legal transitions

Initiated → Authorized → Captured → (Refunded | PartiallyRefunded)
     ↓            ↓
   Failed       Failed

As covered in Section 2's Payment aggregate, a payment's lifecycle is a small, explicit state machine — and the aggregate's own methods (Authorize(), Capture()) are what enforce that only legal transitions are ever possible, throwing rather than silently succeeding if called out of order (attempting to capture a payment that was never authorized, for instance).

Why an explicit state machine matters more here than for most domain objects

Given this guide's emphasis on the cost of a payment-related bug, having every legal and illegal state transition explicitly enumerated and enforced by the aggregate itself — rather than scattered conditional checks across application code — is precisely the kind of rigor this series' DDD guide argues pays for itself most clearly in domains with genuinely complex, high-stakes business rules, and few domains fit that description more clearly than payments.

Terminal states and their permanence

public void Refund(Money refundAmount)
{
    if (Status is not (PaymentStatus.Captured or PaymentStatus.PartiallyRefunded))
        throw new InvalidOperationException($"Cannot refund a payment in status {Status}");
    // ... a Refunded/PartiallyRefunded payment can never transition back to Captured
}

Certain states are genuinely terminal or near-terminal (a Failed payment doesn't transition anywhere further; a fully Refunded payment shouldn't be refundable again) — encoding these as hard constraints in the aggregate is what prevents an entire category of "this should never happen but somehow did" production incidents specific to payment state.

7. Webhooks: Handling Asynchronous Gateway Callbacks

Why payment gateways rely on webhooks, not just synchronous API responses

Your system → gateway.charge() → gateway returns "pending" immediately
    ... (minutes later, potentially) ...
Gateway → POST /webhooks/payment-status → your system, asynchronously reporting the FINAL outcome

Many payment flows (particularly certain card types requiring additional authentication, or bank transfers) don't resolve synchronously within the original API call — the gateway instead sends an asynchronous webhook once the final outcome is known, directly connecting to this series' Event-Driven Architecture guide's core theme: your system needs to handle this exactly like consuming an event from an external, asynchronous source, with all the same discipline (idempotency, per Section 4; ordering awareness) that guide covers for internal messaging.

Verifying webhook authenticity — this is not optional

[HttpPost("/webhooks/payment-gateway")]
public async Task<IActionResult> HandleWebhook()
{
    var payload = await new StreamReader(Request.Body).ReadToEndAsync();
    var signature = Request.Headers["Stripe-Signature"];

    // Verifies the payload genuinely came from the gateway, using a shared secret — per this series'
    // Secret Management and JWT Validation guides' emphasis on never trusting an unverified sender
    var isValid = _gatewaySignatureVerifier.Verify(payload, signature, _webhookSecret);
    if (!isValid) return Unauthorized();

    var evt = ParseWebhookEvent(payload);
    await ProcessWebhookEventAsync(evt);
    return Ok();
}

A webhook endpoint is a publicly reachable URL, by necessity — without verifying the gateway's cryptographic signature on every incoming webhook (using a shared secret, stored per this series' Secret Management guide), an attacker could submit a forged "payment succeeded" webhook and trick your system into believing a payment completed when it never did. This is a direct, concrete application of this series' OWASP Top 10 guide's broken-authentication and injection categories, applied specifically to a payment system's most externally-exposed surface.

Webhook idempotency and out-of-order delivery

if (await _processedWebhookEvents.ExistsAsync(evt.EventId)) return Ok(); // already processed, safe no-op

if (evt.Timestamp < payment.LastUpdatedAt) return Ok(); // an OLDER event arriving late — ignore, don't regress state

As covered in this series' Event-Driven Architecture guide, webhooks are subject to the same at-least-once delivery and potential out-of-order arrival as any other asynchronous message — tracking processed event IDs (idempotency, Section 4 again) and comparing event timestamps against the payment's own last-known state (to avoid a late-arriving, stale webhook incorrectly reverting a payment to an earlier state) are both essential, not optional hardening.

8. The Saga: Coordinating Payment Across Multiple Services

Payment as one step in a larger, cross-service business process

OrderSaga (per this series' Event-Driven Architecture guide):
  1. OrderService: create order (pending)
  2. InventoryService: reserve stock — compensating action: release stock
  3. PaymentService: charge payment — compensating action: REFUND
  4. OrderService: confirm order

As covered directly in this series' Event-Driven Architecture and Microservices guides, "place an order" typically spans multiple services with separate databases — payment is one step in that larger saga, and its compensating action, should a later step fail, is a refund, not a database rollback (since, per Section 1, there's no cross-service ACID transaction spanning the order, inventory, and payment services' separate databases).

Why the compensating action for a payment is itself a genuine, auditable transaction

public async Task CompensateAsync(PaymentId paymentId)
{
    var payment = await _repository.GetByIdAsync(paymentId);
    var refund = payment.Refund(payment.Amount); // a NEW ledger transaction, per Section 3 — never erasing the original charge
    await _gateway.RefundAsync(payment.GatewayCaptureId, payment.Amount);
}

Unlike compensating actions in many other domains (releasing a reserved inventory count, say), a payment's compensation is itself a fully real, ledger-recorded, gateway-executed transaction — this directly reinforces Section 3's append-only ledger principle: a failed downstream step doesn't erase the original charge from history, it records a new, compensating refund transaction alongside it, preserving the complete, honest record of what actually happened.

9. Reconciliation

Why "the ledger looks right" isn't sufficient — it must be proven against external truth

Your ledger says: $48,392.17 captured today
The payment gateway's own settlement report says: $48,392.17 settled today
→ these must match, EXACTLY, every single day

Reconciliation is the (often nightly, automated) process of comparing your system's own ledger against the payment gateway's independently-generated settlement reports, and ultimately against your bank's actual statements — this is the concrete, continuously-enforced verification that Section 1's "money must reconcile" property actually holds, not just an assumption resting on your own system's internal consistency checks alone.

Automating reconciliation, and surfacing discrepancies immediately

public async Task ReconcileAsync(DateOnly date)
{
    var ourRecords = await _ledgerRepository.GetCapturedPaymentsForDateAsync(date);
    var gatewayRecords = await _gateway.GetSettlementReportAsync(date);

    var discrepancies = FindMismatches(ourRecords, gatewayRecords);
    if (discrepancies.Any())
    {
        await _alerting.RaiseAsync("Reconciliation discrepancy detected", discrepancies); // per this series'
                                                                                             // Prometheus/Grafana guide's
                                                                                             // alerting discipline
    }
}

A discrepancy found during reconciliation — a payment your system believes captured that the gateway's settlement report doesn't show, or vice versa — is a genuinely serious signal, treated with the same urgency as a security incident (per this series' OWASP Top 10 guide) rather than a routine data-quality issue to quietly patch; every discrepancy needs to be understood and explained, not merely corrected and forgotten.

Reconciliation as a recurring, automated background process

public class NightlyReconciliationWorker : BackgroundService // per this series' Background Services guide
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // scheduled, per this series' Background Services guide's recurring-job patterns, using Hangfire or Quartz.NET
    }
}

This directly reuses the scheduled background job patterns covered in this series' Background Services guide — reconciliation is precisely the kind of recurring, automated job those patterns are built for, run nightly (or more frequently) without manual intervention, with its own health monitoring (per this series' Health Checks guide's "last successful run" pattern) to ensure the reconciliation job itself hasn't silently stopped running.

10. Fraud and Risk Checks

Where fraud checks fit in the payment flow

Payment request → [Risk scoring: velocity checks, device fingerprinting, address verification]
                        ↓
              Score below threshold: proceed to gateway authorization
              Score above threshold: hold for manual review, or decline outright

A production payment system layers fraud/risk assessment before (or alongside) the actual gateway authorization call — checking transaction velocity (has this card/account attempted an unusual number of payments recently), device and IP reputation, and billing/shipping address consistency, often using a specialized third-party risk-scoring service (analogous to how gateways themselves are typically third-party specialists, per Section 1) rather than building fraud detection from scratch.

The trade-off between fraud prevention and legitimate-customer friction

Every fraud check has a real cost in false positives — a legitimate customer wrongly declined or delayed by an overly aggressive risk check is a genuine, measurable business cost, not a harmless extra precaution; this is a deliberate, ongoing tuning exercise (adjusting risk thresholds based on observed false-positive and fraud-loss rates over time) rather than a "more strict is always better" default.

3D Secure and step-up authentication

Card payment → gateway determines additional authentication is required (3D Secure) →
  customer redirected to their bank's own authentication challenge → returns, payment proceeds

For card payments specifically, 3D Secure (the "Verified by Visa"/"Mastercard Identity Check" flow many customers have encountered) shifts liability for certain fraud disputes from the merchant to the card issuer, in exchange for an additional authentication step — most gateways handle the actual challenge flow, but your system's payment flow (and its state machine, per Section 6) needs to accommodate this additional, asynchronous authentication step as a legitimate part of the payment lifecycle, not an edge case.

11. Data Security and Compliance

PCI DSS: why tokenization (Section 5) is the practical answer, not a checklist to satisfy directly

The Payment Card Industry Data Security Standard (PCI DSS) imposes extensive, genuinely burdensome requirements on any system that stores, processes, or transmits raw card data — the practical, almost universally adopted strategy for a system built on top of a gateway (per Section 5) is to never let raw card data touch your own infrastructure at all, via client-side tokenization, which dramatically narrows your own PCI compliance scope rather than requiring you to build and audit a full PCI-compliant environment yourself.

Encryption and secret management for whatever sensitive data your system does hold

// API keys for the payment gateway itself are exactly the kind of secret covered in this series'
// Secret Management guide — never in source control, ideally via Managed Identity + Key Vault
var gatewayApiKey = await _secretClient.GetSecretAsync("payment-gateway-api-key");

Even with card data itself tokenized away, a payment system still holds genuinely sensitive secrets — gateway API keys, webhook signing secrets — and every principle covered in this series' Secret Management guide applies directly and without exception here: no hardcoded credentials, Managed Identity where the platform supports it, and rotation discipline for anything that could grant an attacker the ability to initiate fraudulent charges or forge webhook events.

Audit logging as a compliance and forensic requirement, not just an operational nicety

logger.LogInformation("Payment {PaymentId} captured for {Amount} by {ActorId}", payment.Id, payment.Amount, actorId);

As covered in this series' Structured Logging and OWASP Top 10 guides, every sensitive action (a payment captured, a refund issued, a risk override applied) needs to be logged with enough context (who, what, when) to support both regulatory audit requirements and forensic investigation after an incident — this is a stricter, more comprehensive logging bar than most systems require, precisely because of Section 1's stakes.

12. Consistency, Availability, and the CAP Trade-off for Money

Why payment systems generally favor consistency over availability, unlike much of this series' general guidance

As covered in this series' System Design guide's CAP theorem discussion, most systems in this series lean toward availability and eventual consistency where possible — a payment system is one of the clearer, most defensible exceptions: it is generally preferable for a payment attempt to fail cleanly (the customer retries, or sees a clear error) than for the system to accept it under uncertain, potentially-inconsistent conditions and risk a ledger discrepancy that reconciliation (Section 9) later has to painstakingly untangle.

Where eventual consistency is still acceptable, deliberately scoped

The LEDGER write (money moved) → strong consistency required, no compromise
A downstream ANALYTICS dashboard showing "today's revenue" → eventual consistency is genuinely fine

Not every part of a payment system needs the same consistency bar — the core ledger write absolutely does, but downstream, read-only projections (a merchant's revenue dashboard, an analytics pipeline) can and should tolerate the same eventual consistency this series' Event-Driven Architecture and CQRS discussions describe generally, since a dashboard being a few seconds stale carries none of the risk a genuinely inconsistent ledger does.

13. Scaling the System

Applying this series' System Design guide's building blocks, with payment-specific emphasis

Read replicas (per this series' SQL Server/PostgreSQL guides): safe for READ-heavy queries
  (transaction history, dashboards) — never route a WRITE that must be immediately consistent to a replica
Caching (per this series' Redis guide): appropriate for relatively static data (merchant configuration,
  fee schedules) — NEVER cache a payment's current status, which must always reflect genuine current state
Queues (per this series' RabbitMQ/Kafka guides): appropriate for the asynchronous parts of the flow
  (webhook processing, sending receipt emails, updating analytics) — NOT for the synchronous
  authorization call itself, which the customer is actively waiting on

Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against this guide's stricter consistency bar (Section 12) before being applied — the general principle "identify the bottleneck, then apply the specific technique" holds, but payments narrow which techniques are safe to apply to which specific part of the flow.

Sharding the ledger, and the partition key that actually matters

Sharding by account_id (or merchant_id) keeps all of one account's ledger entries together,
  making "what is this account's balance" a single-shard query rather than a cross-shard fan-out

As covered in this series' System Design and Cosmos DB/MongoDB guides, choosing the ledger's partition/shard key deliberately — typically the account or merchant ID, since balance queries are the most common and most latency-sensitive access pattern — avoids the expensive cross-shard fan-out that a poorly chosen key (transaction ID, say) would force on every balance check.

14. Observability for a Payment System

Every guide in this series' observability trio, applied with payment-specific stakes

Structured logs (per this series' Structured Logging guide): every payment state transition, logged
  with the payment ID and correlation ID, NEVER logging raw card data or full gateway tokens
Distributed tracing (per this series' Distributed Tracing guide): tracing a single payment's journey
  across the risk check, gateway call, and ledger write — essential for diagnosing where a specific
  slow or failed payment actually got stuck
Metrics (per this series' Prometheus/Grafana guide): payment success rate, gateway latency,
  authorization decline rate — the aggregate health signals a payments team watches continuously

Every technique from this series' observability guides applies directly, with one payment-specific addition worth stating explicitly: logs and traces must never capture raw card numbers, full gateway tokens, or CVV data, even for debugging purposes — this is a hard, non-negotiable line directly extending this series' OWASP Top 10 and Secret Management guides' "never log sensitive data" principle, applied here with genuinely higher stakes than almost any other domain in this series.

Alerting on payment-specific symptoms

# Per this series' Prometheus/Grafana guide's symptom-based alerting principle, applied to payments
rate(payment_declined_total[5m]) / rate(payment_attempted_total[5m]) > 0.15  # a sudden decline-rate spike

A sudden spike in the payment decline rate, or in gateway latency, is exactly the kind of user-facing symptom this series' Prometheus/Grafana guide argues alerts should be built around — and for a payment system, the on-call response to such an alert carries unusually direct business consequences (lost revenue, frustrated customers), which is precisely why this category of alert deserves genuinely fast, well-rehearsed incident response.

15. Common Pitfalls

Quick Reference Table

Conclusion

A payment processing system takes every general system design technique covered throughout this series and applies it under a stricter, less forgiving correctness bar — because the cost of a bug here is measured in real money moved incorrectly, not just degraded user experience. The design that actually holds up under that bar rests on a small number of non-negotiable foundations: a double-entry, append-only ledger as the provable source of truth; idempotency enforced at every single layer a payment touches; an explicit, aggregate-enforced state machine governing what transitions are even possible; and continuous, automated reconciliation that treats any discrepancy as a genuine incident rather than a rounding error to quietly absorb.

Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — DDD's aggregates enforcing business rules, Event-Driven Architecture's sagas and idempotent consumers, Secret Management's discipline around gateway credentials, and the full observability trio watching over a system where "we'll notice eventually" is never an acceptable answer. Payments are, in that sense, less a distinct discipline from everything else in this series than the place where its cumulative lessons about correctness, idempotency, and honest reconciliation with reality matter more visibly and more unforgivingly than almost anywhere else.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the reconciliation discrepancy that turned out to matter far more than a rounding error ever should.