Skip to content
akshay
All work

Practice Reliability engineering

Reliability patterns for enterprise integration

Four patterns decide whether an integration survives its dependencies, and each one buys something at a price worth stating. They are drawn here as they behave rather than as they are defined. Retry and dead-lettering carried the two integration case studies; idempotent processing is designed into the platform; circuit breaking is the one I reach for at an API boundary shared by several callers.

The patterns

Retry with backoff

01

A transient fault: the endpoint was briefly unreachable, throttled, or slow enough to time out. The same request would succeed a moment later.

Scroll the diagram sideways

A caller retries a failing service three times, with the interval growing between each attempt, up to a fixed attempt limit. attempt 1 · immediate attempt 2 · +2s attempt 3 · +4s, then stop Caller idempotent op Service transient fault add jitter, or every caller retries on the same schedule
Three attempts, each further apart than the last, then stop.
When to reach for it
When the operation is idempotent and the fault is genuinely transient. Bounded, always — an attempt limit and a growing interval.
What it costs
Latency amplification under load, and a retry storm if every caller backs off on the same schedule. Add jitter, or the retries synchronise and become the outage.

Dead-lettering

02

A message that will never succeed: malformed payload, a contract that changed, a reference to something that no longer exists. Retrying it forever is just load.

Scroll the diagram sideways

A queue feeds a handler that retries within a budget; once that budget is spent the message is routed to a dead-letter queue, which raises an alert to a named owner. budget spent alert bounded retry Queue Handler n attempts Dead letter Owner runbook
The retry budget ends somewhere, and that somewhere needs an owner.
When to reach for it
Immediately after the retry budget is exhausted. The queue needs a named owner, an alert and a runbook before it is switched on.
What it costs
An operational obligation. A dead-letter queue nobody watches is strictly worse than no dead-letter queue, because it looks like handling.

Circuit breaking

03

A dependency that is down rather than slow. Every caller is queueing against it, holding connections, and the retries are making the recovery slower.

Scroll the diagram sideways

A circuit breaker moves from closed to open once a failure threshold is crossed, to half-open after a cool-down, and then back to closed if a probe succeeds or to open if it fails. failure threshold cool-down elapsed probe fails probe succeeds Closed calls pass Open fail fast Half-open one probe
Three states. The half-open probe is the part people leave out.
When to reach for it
In front of a downstream that can be overwhelmed, especially one shared by several callers. Needs a half-open probe to discover recovery.
What it costs
It fails requests that might have succeeded, and it moves the failure to the caller, who now needs a fallback. Tuning the thresholds is genuinely hard.

Idempotent processing

04

At-least-once delivery doing exactly what it promises: the same message arriving twice, or a third party replaying yesterday because their side had an incident.

Scroll the diagram sideways

A message and its redelivery both carry the same business key. The handler applies the effect for the first and discards the second, so the effect happens exactly once. Message key K Redelivery key K Handler seen(K)? Effect applied once Discarded no second effect
Two deliveries, one business key, one effect.
When to reach for it
Always, in a messaging estate. Key on a business identifier, not on a delivery or message id, which changes on redelivery.
What it costs
Every handler carries it, and there is no central enforcement — it lives or dies on code review. Storage for the seen-keys record, and a decision about how long to keep it.

Where these show up

Retry and dead-lettering carried both the ERP integration layer and the estate rescue, where redesigning the two together — bounded, owned, alerted — was my part of what moved the incident rate; idempotent processing is keyed on a business identifier in both, and all four are what the migration target patterns assume of anything built to them. Circuit breaking is the one I reach for at an API boundary shared by several callers, and it is the pattern neither of those two pieces of work needed — though it is the one a colleague put in front of a fixed SOAP layer during the storm that first showed me a downstream failing under load. The longer argument is in Designing for the third party having a bad day.

All work