Choosing Retry Budgets and Deadline Propagation

Retries are the mechanism that makes idempotency necessary, and unbounded retries are the mechanism that turns a degraded dependency into a full outage. This runbook covers the two controls that bound them — a client-wide retry budget and a propagated deadline — and sits under retry logic & backoff fundamentals.

Prerequisites. Your write endpoints must already be safely retryable, per API contract design for idempotent endpoints — a retry budget on a non-idempotent endpoint just bounds how many duplicates you create. You should also have backoff with jitter in place, since a budget controls volume while backoff controls timing.

Step 1 — Measure your amplification factor

Every hop that retries independently multiplies the load its dependencies see. The arithmetic is unforgiving.

Retry amplification through a three-hop call chain One user request enters the edge service. The edge retries up to three times against the orders service, which retries up to three times against the ledger service, which retries up to three times against the database. The call counts compound: one, three, nine, twenty-seven. A note contrasts this with a retry budget capping total retries at ten percent of successful volume. edge 1 ×3 orders 3 ×3 ledger 9 ×3 database 27 calls With a 10% retry budget per hop: 1 → 1.1 → 1.21 → 1.33 calls Amplification collapses from 27x to 1.33x — the same transient failures absorbed, without the load spike that converts a slow dependency into a dead one.

The failure is not gradual. A leaf service running at 70% capacity that slows down slightly causes its callers to time out; the callers retry; the leaf now sees 3x load at 210% of capacity and stops responding entirely. Retries did not respond to the outage — they caused it.

# Your actual amplification factor, per dependency. Anything above 1.2 is a risk.
sum(rate(http_client_requests_total{peer="ledger"}[5m]))
  / sum(rate(http_server_requests_total{service="orders",code=~"2.."}[5m]))

Step 2 — Replace attempt counts with a budget

A per-call max_attempts=3 is a local decision with global consequences: it guarantees 3x load in the worst case regardless of how healthy the system is. A budget is a global decision: retries are permitted only while their recent volume stays under a fraction of successful volume.

import time, threading

class RetryBudget:
    """Token bucket sized as a ratio of successful requests.

    ratio=0.1 permits one retry per ten successes; min_per_sec keeps a low-traffic
    client from being unable to retry at all.
    """
    def __init__(self, ratio: float = 0.1, ttl_s: float = 10.0, min_per_sec: float = 3.0):
        self.ratio, self.ttl_s, self.min_per_sec = ratio, ttl_s, min_per_sec
        self._tokens, self._updated = 0.0, time.monotonic()
        self._lock = threading.Lock()

    def _decay(self) -> None:
        now = time.monotonic()
        elapsed = now - self._updated
        self._updated = now
        # Tokens age out over the window so a burst of successes cannot bank credit.
        self._tokens *= max(0.0, 1.0 - elapsed / self.ttl_s)

    def record_success(self) -> None:
        with self._lock:
            self._decay()
            self._tokens += self.ratio

    def try_withdraw(self) -> bool:
        with self._lock:
            self._decay()
            floor = self.min_per_sec * self.ttl_s * self.ratio
            if self._tokens + floor < 1.0:
                return False              # budget exhausted — fail fast, do not retry
            self._tokens -= 1.0
            return True
def call_with_budget(fn, budget: RetryBudget, deadline: float, key: str):
    attempt = 0
    while True:
        try:
            result = fn(idempotency_key=key, timeout=remaining(deadline))
            budget.record_success()
            return result
        except Transient as exc:
            attempt += 1
            if not budget.try_withdraw():
                metrics.inc("retry_budget_exhausted_total")
                raise                      # the budget said no: this is correct, not a bug
            delay = backoff(attempt)
            if time.monotonic() + delay > deadline:
                raise DeadlineExceeded from exc
            time.sleep(delay)

The important property: when the dependency is healthy, successes accrue tokens and retries are freely available. When it is failing, successes stop, tokens decay, and retries shut off automatically — exactly when they would do the most damage.

Step 3 — Propagate a deadline, not a timeout

Per-hop timeouts compound in the same way retries do. Five services with a five-second timeout each permit twenty-five seconds of work on behalf of a caller who gave up at three.

// Edge: set an absolute deadline once, from the user-facing SLO.
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()

// Every outbound call carries the remaining time, not a fresh constant.
req.Header.Set("X-Request-Deadline", strconv.FormatInt(deadlineUnixMillis(ctx), 10))
// Downstream: adopt the caller's deadline, never extend it.
func DeadlineMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        if hdr := r.Header.Get("X-Request-Deadline"); hdr != "" {
            if ms, err := strconv.ParseInt(hdr, 10, 64); err == nil {
                d := time.UnixMilli(ms)
                if remaining := time.Until(d); remaining <= 0 {
                    http.Error(w, "deadline already exceeded", 504)
                    return                      // do not start work nobody will read
                }
                var cancel context.CancelFunc
                ctx, cancel = context.WithDeadline(ctx, d)
                defer cancel()
            }
        }
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

gRPC does this natively — grpc-timeout is propagated and decremented by every hop — which is one of the strongest practical arguments for it in a deep call graph. Over plain HTTP you have to build it, and the header must carry an absolute instant, not a duration, so that network transit time is charged against the budget rather than silently forgiven at each hop.

Step 4 — Refuse retries that cannot fit

Deadline-aware retry admission A three-second deadline drawn as a horizontal bar. The first attempt consumes eight hundred milliseconds and fails, followed by a two hundred millisecond backoff. The second attempt consumes nine hundred milliseconds and fails, followed by a four hundred millisecond backoff. Only seven hundred milliseconds remain, which is less than the observed p99 attempt duration, so the third attempt is refused and a deadline-exceeded error is returned immediately. t=0 t=3000 ms — deadline attempt 1 · 800 ms · fail 200 attempt 2 · 900 ms · fail 400 700 ms left < p99 attempt (900 ms) attempt 3 REFUSED return 504 now, free the connection Rule: retry only if now + backoff + p99_attempt < deadline Starting an attempt that cannot finish burns a connection, a worker and a downstream slot for a result nobody will ever read. 
def may_retry(deadline: float, backoff_s: float, p99_attempt_s: float) -> bool:
    return time.monotonic() + backoff_s + p99_attempt_s < deadline

Feed p99_attempt_s from the client’s own observed latency histogram rather than a constant. A dependency that has slowed from 200 ms to 2 s should automatically stop being retried, and a hard-coded estimate will not notice.

Step 5 — Alert on exhaustion

Signal Healthy Action threshold What it means
retry_budget_utilisation under 0.3 above 0.8 for 5 min Retries are close to being shut off; the dependency is degrading
retry_budget_exhausted_total 0 any sustained rate Retries are already off — errors are now reaching users directly
retry_amplification_factor 1.0–1.1 above 1.3 Some hop is retrying outside the budget
deadline_exceeded_total under 0.1% above 1% End-to-end budget is too tight, or a hop is not propagating
deadline_header_present_ratio above 0.99 below 0.95 A service or proxy is dropping the deadline header

Do not alert on raw retry count. A high retry rate during a brief blip is the system working; a budget at 90% utilisation is the system about to stop working.

Verification and testing

Confirm the budget actually closes. Fail a dependency and assert that retries stop rather than continuing at 3x.

def test_budget_shuts_retries_off(fake_dep, budget):
    for _ in range(100):
        call_with_budget(fake_dep.ok, budget, deadline=far_future(), key=new_key())

    fake_dep.fail_all()
    attempts_before = fake_dep.call_count
    for _ in range(200):
        with contextlib.suppress(Exception):
            call_with_budget(fake_dep.call, budget, deadline=far_future(), key=new_key())

    extra = fake_dep.call_count - attempts_before
    assert extra < 200 * 1.2      # not 200 * 3 — the budget capped the amplification

Confirm the deadline reaches the last hop.

curl -s -H 'X-Request-Deadline: '$(( ($(date +%s) + 3) * 1000 )) \
     -H 'X-Debug-Echo-Deadline: 1' http://edge/v1/orders/probe | jq
# expect: every hop reports a deadline within a few ms of the value sent

Confirm work is abandoned when the deadline passes.

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  -H 'X-Request-Deadline: '$(( ($(date +%s) - 1) * 1000 )) http://edge/v1/orders/probe
# expect: 504 in well under 0.05s — no work started

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
A leaf service degrades slightly; three layers of max_attempts=3 turn it into a 27x load spike and a full outage Replace fixed attempt counts with a shared budget in the client library, and roll it out from the deepest hop outward so the innermost amplification is removed first. retry_amplification_factor per dependency; page above 1.3
Deadline header is set at the edge but a middle service creates a fresh 5-second timeout, so downstream work outlives the client Adopt-never-extend as a library invariant: the middleware may only shorten. Add a contract test asserting the propagated deadline is monotonically non-increasing along the chain. deadline_header_present_ratio per service; a trace attribute carrying the deadline at every span
Budget is exhausted during a genuine incident and errors reach users, and the team disables the budget to “restore retries” Treat exhaustion as the budget working. Raise the ratio deliberately with evidence, or fix the dependency; do not remove the cap during the incident it is protecting you from. retry_budget_exhausted_total alongside dependency error rate on one dashboard, so cause and effect are visible together
Low-traffic clients cannot retry at all because 10% of two successes per minute is less than one token Set a min_per_sec floor, as in the implementation above, so sparse callers keep a small absolute allowance. retry_budget_utilisation by client; a value pinned at 1.0 for low-QPS clients indicates a missing floor
Retries are budget-capped but not idempotent, so the surviving retries create duplicate charges Fix the endpoint contract first. A budget limits how many duplicates you create; it does not prevent them. See API contract design for idempotent endpoints. Effect rows per distinct idempotency key — must stay at exactly 1.0
The budget closes itself exactly when it should A chart of available retry tokens over time. While the dependency is healthy, successes accrue tokens and the level stays high. When the dependency starts failing, successes stop, existing tokens decay, and the level reaches zero within seconds — so retries shut off automatically at the moment they would do the most damage. tokens dependency starts failing successes accrue tokens budget exhausted — retries off No operator action, no threshold to tune: no successes means no retry credit.

SRE / observability checklist

  1. retry_budget_utilisation — Gauge of tokens consumed over tokens available, per dependency. Alert above 0.8 for 5 minutes; it is the earliest signal of a degrading dependency.
  2. retry_budget_exhausted_total — Counter of refused retries. Any sustained rate means user-visible errors are no longer being absorbed.
  3. retry_amplification_factor — Derived ratio of outbound calls to inbound successes per hop. Page above 1.3.
  4. deadline_exceeded_total — Counter by service. Above 1% of requests means the end-to-end budget is unrealistic for the call graph.
  5. deadline_header_present_ratio — Gauge per service. Below 0.95 means a hop or proxy is dropping the header and downstream work is unbounded.
  6. request.deadline_ms span attribute — On every span, so a trace shows exactly where the budget was consumed and which hop abandoned it.