Availability vs Reliability vs Durability (Why a System Can Be Up and Still Lose Your Data)

A checkout service answers every request in 20 milliseconds, its health checks are green, and its uptime dashboard reads 99.99% for the quarter. Yet a customer has been charged twice, and three orders placed just before a database failover no longer exist. By the metric the team watches most closely, nothing went wrong.

The contradiction disappears once you separate three guarantees that are often collapsed into the single word "reliable." Availability is the promise that the system answers. Reliability is the promise that the answer is correct. Durability is the promise that data the system said it saved is still there later. These guarantees are produced by different mechanisms, measured with different signals, and frequently traded against one another.

Understanding availability vs reliability vs durability is less about vocabulary and more about failure analysis. Each guarantee lives at a different place in the system: availability on the request path, reliability in the semantics of operations, durability in how state is persisted and replicated. This article examines how each one is measured, which architectural mechanisms improve it, and how improving one can quietly degrade another.

Availability vs Reliability vs Durability: Three Questions About One System

The simplest way to separate the three guarantees is to ask what question each one answers.

A note on terminology: in SRE practice, "reliability" is often used as an umbrella covering all of these. That usage is fine for organizational goals, but it hides the distinctions that matter during design. Here, reliability means correct behavior, which is a stronger property than responding.

These properties are independent because they depend on different parts of the system. Availability depends on whether some healthy path exists between the client and a component that can respond. Reliability depends on whether operations preserve their invariants under retries, concurrency, and partial failure. Durability depends on where state physically lives at the moment of acknowledgment and how many independent failures it can survive.

A concrete causal chain

Consider a hypothetical checkout API. It writes orders to a primary database that replicates asynchronously to a standby, then publishes an event to a message broker for fulfillment.

Now the database becomes slow rather than unavailable, perhaps because of lock contention or a saturated disk. The causal chain unfolds like this:

slow database → queries hold connections longer → connection pool exhausts → API requests queue waiting for connections → latency rises past client timeouts → clients retry → retries add load to the already slow database → more timeouts

At this stage, availability is degrading. But something subtler has happened. A client that timed out does not know whether its write committed. Some of those retried requests succeed a second time, and without idempotency, the customer now has two orders. Reliability has been violated, even though every individual response was a success.

Then the operators fail over to the standby. Because replication was asynchronous, the standby is a few seconds behind. Orders that the primary acknowledged but had not yet shipped to the standby are gone. Durability has been violated, and the system came back up quickly, so availability looks excellent in the postmortem metrics.

One incident, three distinct failures, and only one of them shows up on an uptime dashboard.

Distributed Systems Deep Dive: The Mechanisms Behind Each Guarantee

Availability: redundancy, failure detection, and dependency math

The mental model for availability is paths. A request is available if at least one complete path of healthy components exists from client to response within the deadline.

Serial dependencies multiply. If a request synchronously depends on five services, each available 99.9% of the time with independent failures, the request's availability is roughly 0.999⁵ ≈ 99.5%. That turns about 43 minutes of monthly downtime per service into roughly 3.6 hours for the composite request. Every synchronous dependency you add is a tax on availability.

Redundancy is the counterforce. Two independent replicas, each 99% available, give 1 − (0.01)² = 99.99% if either one can serve the request. The word that does the work in that formula is independent. Replicas that share a deployment pipeline, a configuration system, a kernel version, a power domain, or a region fail together. Correlated failure is why redundancy often delivers far less availability than the arithmetic promises.

Redundancy also depends on failure detection, which is fundamentally uncertain. A timeout cannot distinguish a dead node from a slow one or from a network partition. Effective downtime during a failover is detection time plus promotion time plus the time clients take to discover the new target. In practice, availability is often dominated by MTTR rather than MTBF: systems that fail occasionally but recover in seconds outperform systems that fail rarely but recover in an hour.

The new complexity redundancy introduces is state. Stateless replicas are easy to make redundant. Stateful replicas need a story for which copy is authoritative, and that story is where durability and reliability start to compete with availability.

Reliability: correctness under retries and partial failure

Reliability asks whether the system preserves its invariants: an order is created exactly once per customer intent, an account balance never goes negative, an inventory count matches reservations.

The central threat is the ambiguous outcome. When a client times out, the operation may have failed, succeeded, or still be in progress. Retries convert that ambiguity into availability: the client gets a response eventually. But retries convert ambiguity into duplicates unless the operation is idempotent.

A common pattern is to store an idempotency key in the same transaction as the side effect, so the key and the write commit or fail together:

func (s *OrderService) PlaceOrder(ctx context.Context, req PlaceOrderRequest) (string, error) {
	tx, err := s.db.BeginTx(ctx, nil)
	if err != nil {
		return "", err
	}
	defer tx.Rollback() // no-op after a successful Commit

	// Claim the key inside the same transaction as the order write.
	// A concurrent duplicate blocks on the unique index until this commits.
	res, err := tx.ExecContext(ctx,
		`INSERT INTO idempotency_keys (key, order_id) VALUES ($1, $2)
		 ON CONFLICT (key) DO NOTHING`,
		req.IdempotencyKey, req.NewOrderID)
	if err != nil {
		return "", err
	}
	if n, _ := res.RowsAffected(); n == 0 {
		// Duplicate request: return the original result, create nothing new.
		var existing string
		err := s.db.QueryRowContext(ctx,
			`SELECT order_id FROM idempotency_keys WHERE key = $1`,
			req.IdempotencyKey).Scan(&existing)
		return existing, err
	}

	if err := insertOrder(ctx, tx, req); err != nil {
		return "", err
	}
	// Transactional outbox: the event is persisted atomically with the order,
	// so a broker outage cannot produce an order without an event.
	if err := insertOutboxEvent(ctx, tx, "order.placed", req); err != nil {
		return "", err
	}
	return req.NewOrderID, tx.Commit()
}

Two details matter. First, the key must be bound to the request content (a hash of the payload), otherwise a client bug that reuses a key for a different order silently returns the wrong result. Second, the outbox exists because publishing to a broker after commit is a dual write: the process can crash between the two, and the system ends up with an order that fulfillment never hears about. The outbox moves that problem into a relay that retries publication, which in turn requires consumers to deduplicate. Complexity has been moved, not eliminated, but it has been moved somewhere it can be handled systematically.

Reliability is also why availability tricks deserve scrutiny. Serving a stale cached price when the pricing service is down raises availability and lowers correctness. Whether that is acceptable is a product decision disguised as an infrastructure setting.

Durability: where is the acknowledgment point?

The mental model for durability is the acknowledgment point: at the moment the system says "saved," where does the data physically exist, and how many independent failures would destroy every copy?

A database that acknowledges a write after placing it in the OS page cache is fast and available but loses data on power loss. Acknowledging after an fsync of the write-ahead log survives a process or machine crash but not a disk failure. Acknowledging after synchronous replication to a second machine survives a machine loss. Acknowledging after a quorum write across failure domains survives the loss of a zone.

Each step up costs latency, and the replication steps cost availability. With synchronous replication, if the replica is unreachable, the primary must either block writes (preserving durability, sacrificing availability) or degrade to asynchronous mode (preserving availability, silently sacrificing durability). Systems that make this switch automatically should make it loudly visible, because the durability guarantee has changed while every request still succeeds.

Quorum systems formalize the tradeoff. With N replicas, writing to W and reading from R where W + R > N ensures that a read quorum overlaps the most recent write quorum. Larger W improves durability and lowers write availability, because more replicas must be reachable to accept a write.

Replication protects against hardware loss but not logical loss. A bug or an operator command that deletes rows is replicated faithfully to every copy within milliseconds. Durability against that class of failure requires point-in-time recovery, versioned objects, or immutable backups held in a separate failure domain with separate credentials.

At scale, durability becomes a rate problem. With thousands of disks, failures are continuous. Durability depends on whether the system re-replicates lost copies faster than additional failures occur. Some object stores advertise durability on the order of eleven nines per object per year; figures like that are design targets under modeled independent failures, not protection against software bugs or human error.

Architecture: Mapping Each Guarantee to a Component

Consider an example order platform designed so that each guarantee has a clear owner.

Clients (send Idempotency-Key header)
  |
  v
Load Balancer  -- health checks, removes failed API nodes (availability)
  |
  v
API Service (stateless, N replicas across zones)  -- availability
  |
  v
Primary DB ----sync----> Standby (other zone)    -- durability (RPO = 0 within region)
  |   \
  |    \--async--------> Replica (other region)  -- disaster recovery (RPO > 0)
  |
  | outbox table, committed with the order        -- reliability
  v
Outbox Relay --> Message Broker --> Fulfillment Workers (dedupe by event ID)
  |
  v
Continuous WAL archive + snapshots in isolated storage  -- durability vs logical loss

Request flow. The load balancer routes to any healthy stateless API node. API nodes hold no authoritative state, so losing one costs only in-flight requests, which clients retry safely because of idempotency keys.

State ownership. The primary database is the single source of truth for orders. The broker holds derived events; if it loses messages, the outbox can republish. The cross-region replica is a recovery asset, not a serving path, because its asynchronous lag means promoting it can lose recent writes.

Failure boundaries. An API node failure is an availability event with near-zero impact. A broker outage delays fulfillment but does not block checkout, because the outbox decouples order acceptance from event delivery. A primary failure triggers promotion of the synchronous standby: a brief availability dip, no durability loss. A regional failure forces a choice: promote the async replica and accept a bounded loss of acknowledged orders, or wait for the region and accept extended unavailability.

Consistency boundary. Everything inside the primary transaction (order, idempotency key, outbox event) is atomic. Everything downstream is eventually consistent and must tolerate duplicates and delays.

Bottleneck. Synchronous replication puts cross-zone round-trip latency on every commit. Write throughput is bounded by the primary and that round trip; the stateless tier can scale horizontally, but it cannot scale past the serialized write path without partitioning the data.

The design's key property is that the regional failover decision is explicit. The architecture does not pretend to deliver zero RPO and full availability across regions simultaneously; it names the tradeoff and assigns it to a runbook.

Availability vs Reliability vs Durability Tradeoffs: A Decision Framework

Most mechanisms improve one guarantee by spending another.

The useful decision question is not "which is most important?" but "what does each kind of failure cost for this specific data?"

• No answer: the cost of unavailability. For a social feed, it is high relative to the alternatives; users notice a blank feed more than a slightly stale one.
• Wrong answer: the cost of unreliability. For a payment ledger, a duplicate charge is worse than a failed request the client can retry.
• Lost answer: the cost of nondurability. For an order record, losing an acknowledged write breaks a promise to a customer and can be worse than refusing the order outright.

These priorities differ within a single product. A checkout system might favor durability and correctness for the order write, availability for product browsing, and availability with staleness for recommendations. Treating the whole system with one policy usually means overpaying for some data and underprotecting other data.

Error budgets make these tradeoffs operational. An availability SLO of 99.9% grants a budget for unavailability; a correctness SLO (for example, zero tolerated duplicate charges, verified by reconciliation) grants none. Writing both down forces the discussion about which guarantee yields first under stress.

Distributed Systems Failure Modes at Scale

The relevant failure modes change as load grows. The numbers below are illustrative.

At 10 requests/second, a single primary with a synchronous standby, an idempotent write path, and tested backups covers most needs. Retry storms are unlikely because there is ample headroom. Adding multi-region active-active here buys complexity and conflict-resolution risk for availability the business may not need.

At 1,000 requests/second, overload dynamics dominate. A dependency slowdown fills connection pools in seconds, and naive retries can multiply load by the retry count at exactly the wrong moment. Retry budgets, jittered backoff, deadlines propagated through the call graph, and load shedding become availability requirements. Duplicates stop being theoretical; idempotency becomes a reliability requirement.

At 100,000 requests/second, the single write primary is likely the bottleneck, so data is partitioned. Partitioning improves throughput and narrows failure blast radius, but it introduces cross-partition invariants that no single transaction can protect. Durability turns into a repair-rate problem: a lost node triggers re-replication traffic that competes with production load, and a correlated event (a bad deploy, a zone outage) can trigger many repairs at once.

Across all scales, the most dangerous failures are silent transitions: replication falling back to async, a cache serving stale data longer than intended, a consumer silently skipping poison messages. Each preserves availability while degrading another guarantee without any user-visible error.

Engineering Principles

1. A successful response is not a correct response. Availability metrics measure that the system answered, not that the answer preserved its invariants. Correctness needs its own signals: reconciliation, invariant checks, and duplicate detection.

2. Durability is defined at the acknowledgment point. Whatever survives the failures possible at the moment of "saved" is the true durability guarantee, regardless of what the architecture diagram implies.

3. Retries buy availability with correctness and capacity. Without idempotency, retries create duplicates; without budgets, they amplify overload.

4. Redundancy only helps against independent failures. Replicas that share configuration, code, credentials, or failure domains fail together, and the availability math no longer applies.

5. Replication is not backup. Replication copies mistakes as faithfully as data. Protection against logical loss requires history that a bad command cannot reach.

6. Every degraded mode should be visible. Fallback to async replication, stale reads, or skipped messages trade one guarantee for another. A system that makes that trade silently has made it on behalf of people who never agreed to it.

7. Guarantees belong to data, not to systems. Different data in the same product warrants different tradeoffs, and a single policy applied uniformly is usually wrong somewhere.

Final Thoughts

The distinction between availability vs reliability vs durability matters because each guarantee fails in a different place and announces itself differently. Unavailability is loud: errors and timeouts. Unreliability is quiet: duplicate charges and drifted balances discovered days later. Nondurability is often invisible until someone looks for data that should exist.

The deeper lesson is that a system's guarantees are determined by its behavior under stress, not by its behavior when healthy. A healthy system looks available, reliable, and durable all at once. The real design is revealed when a dependency slows, a replica falls behind, or a client retries a request whose outcome it cannot know. At those moments, the architecture has to choose which promise to keep, and good engineering means that choice was made deliberately, per class of data, with its costs written down.

These are the kinds of problems at the center of modern software systems and the work PulseSoft focuses on: understanding where state lives, which promises each component makes, and what happens when the assumptions between them stop being true.

Key Takeaways

• Availability measures whether the system responds, reliability measures whether it responds correctly, and durability measures whether acknowledged data survives; a system can score well on one while failing another.
• Serial synchronous dependencies multiply unavailability, and redundancy only offsets it when replica failures are truly independent.
• Timeouts create ambiguous outcomes, so retries improve availability only when operations are idempotent; otherwise they convert ambiguity into duplicate side effects.
• The durability guarantee is whatever survives at the acknowledgment point; asynchronous replication means some acknowledged writes can be lost on failover.
• Synchronous replication and quorum writes improve durability by spending write latency and write availability, especially during partitions.
• Replication protects against hardware loss but propagates logical errors instantly, so point-in-time recovery and isolated backups are separate durability mechanisms.
• The right tradeoff depends on the cost of no answer, a wrong answer, or a lost answer for each class of data, and degraded modes that change those guarantees should always be observable.