Measuring and Bounding NTP Clock Skew in Production

Every TTL, lease and expiry check in a distributed system rests on an assumption about how far apart the clocks are. Most teams never write the assumption down, let alone monitor it — which means the first time it is tested is during an incident. This runbook turns it into a number with an alert on it, implementing the measurement half of clock skew & time ordering in distributed systems.

Prerequisites. Root or configuration-management access to the hosts running your deduplication path, and a metrics pipeline that can scrape a node exporter or accept application metrics.

Step 1 — Configure chrony to slew, not step

# /etc/chrony/chrony.conf
pool time.cloudflare.com iburst maxsources 4
server 169.254.169.123 iburst prefer     # cloud provider's local NTP, lowest latency

<svg viewBox="0 0 760 165" role="img" aria-label="Slewing versus stepping shown as two correction curves, one smooth and monotonic and one discontinuous" xmlns="http://www.w3.org/2000/svg" style="width:100%;max-width:740px;height:auto;display:block;margin:1.5rem auto;">
  <rect class="svg-bg" x="0" y="0" width="760" height="165" fill="var(--color-parchment)"/>
  <title>Slew keeps time monotonic; step does not</title>
  <desc>Two correction curves for the same offset. Slewing runs the clock slightly fast until the offset closes, producing a smooth monotonic line. Stepping jumps the clock instantly, producing a discontinuity during which a live key can appear expired and a held lease can appear to have decades left.</desc>
  <line x1="60" y1="130" x2="720" y2="130" stroke="currentColor" stroke-width="1.3"/>
  <line x1="60" y1="130" x2="60" y2="26" stroke="currentColor" stroke-width="1.3"/>
  <path d="M62,110 C200,104 320,72 460,50 C560,38 650,34 716,32" fill="none" stroke="currentColor" stroke-width="2.4"/>
  <text x="470" y="76" font-size="10.5" font-weight="bold" fill="currentColor">slew — monotonic, converges slowly</text>
  <path d="M62,110 L300,106" fill="none" stroke="currentColor" stroke-width="2.2" stroke-dasharray="6 4"/>
  <path d="M300,40 L716,36" fill="none" stroke="currentColor" stroke-width="2.2" stroke-dasharray="6 4"/>
  <line x1="300" y1="106" x2="300" y2="40" stroke="currentColor" stroke-width="1.4" stroke-dasharray="3 3"/>
  <text x="120" y="52" font-size="10.5" font-weight="bold" fill="currentColor">step — instant, discontinuous</text>
  <text x="312" y="126" font-size="9.5" font-weight="bold" fill="currentColor">every TTL comparison across this jump is wrong</text>
</svg>

# Step ONLY during the first three updates after boot; slew for ever after.
# A step during normal operation is what expires live idempotency keys early.
makestep 1.0 3

# Slew aggressively enough to converge, gently enough to stay monotonic.
maxslewrate 100

driftfile /var/lib/chrony/chrony.drift
rtcsync
logdir /var/log/chrony
log tracking measurements statistics

sudo systemctl restart chrony
chronyc tracking          # System time / Last offset / RMS offset
chronyc sources -v        # which sources are selected and their reachability

Slewing corrects an offset by running the clock slightly fast or slow rather than jumping it. That keeps wall time monotonic, which is what every TTL comparison quietly assumes. The cost is slower convergence after a large offset, which is why makestep is retained for boot.

Step 2 — Export the offset

#!/usr/bin/env bash
# /usr/local/bin/clock-metrics.sh — run every 30 s via systemd timer.
set -euo pipefail
OUT=/var/lib/node_exporter/textfile/clock.prom
TMP="$OUT.$$"

TRACK=$(chronyc -c tracking)                       # CSV: ref,addr,stratum,...
OFFSET=$(echo "$TRACK" | cut -d, -f5)              # last offset, seconds
RMS=$(echo "$TRACK"    | cut -d, -f7)              # RMS offset, seconds
STRATUM=$(echo "$TRACK"| cut -d, -f3)

cat > "$TMP" <<EOF
# HELP node_clock_offset_seconds Last measured offset from the NTP reference.
# TYPE node_clock_offset_seconds gauge
node_clock_offset_seconds $OFFSET
# HELP node_clock_rms_offset_seconds RMS offset over the recent window.
# TYPE node_clock_rms_offset_seconds gauge
node_clock_rms_offset_seconds $RMS
# HELP node_clock_stratum NTP stratum of the selected source.
# TYPE node_clock_stratum gauge
node_clock_stratum $STRATUM
EOF
mv "$TMP" "$OUT"          # atomic replace — the exporter never reads a partial file
# Also record steps, which should be rare and are always worth investigating.
journalctl -u chrony --since '1 hour ago' | grep -c 'System clock was stepped'

For managed runtimes where chrony is not reachable, measure at the application layer against a store that stamps its own time:

-- The database's clock is the reference; the application's is the sample.
SELECT extract(epoch FROM (now() - $1::timestamptz)) AS offset_seconds;
sample = datetime.now(timezone.utc)
offset = conn.execute("SELECT extract(epoch FROM (now() - %s))", (sample,)).scalar()
clock_offset_gauge.set(float(offset))     # includes round-trip time; halve it if precise

Step 3 — Derive the fleet spread

Why the spread, not the average, is the number that matters A horizontal axis of clock offset in milliseconds with zero in the centre. Nine host markers are plotted, most clustered near zero, one at minus thirty-eight milliseconds and one at plus forty-two milliseconds. The average is close to zero and misleading. A bracket spans the two extremes and is labelled fleet spread of eighty milliseconds, annotated as the real bound on the unsafe window. 0 ms −50 +50 −38 7 hosts within ±5 ms +42 fleet spread = 80 ms this is the unsafe window for any off-store TTL mean offset ≈ +1 ms — reassuring, and completely irrelevant 
# THE metric. Not the mean, not the max — the disagreement between extremes.
max(node_clock_offset_seconds) - min(node_clock_offset_seconds)

# Per role, so a skewed batch fleet does not mask a healthy payments fleet.
max by (role) (node_clock_offset_seconds) - min by (role) (node_clock_offset_seconds)

# The 99.9th percentile over 30 days is the number to build TTL padding from.
quantile_over_time(0.999,
  (max(node_clock_offset_seconds) - min(node_clock_offset_seconds))[30d:1m])

Scope the spread to hosts that actually interact. A batch-processing fleet 200 ms out matters only if it reads keys written by the API fleet; if it does not, folding it into one global metric produces alerts nobody can act on.

Step 4 — Set thresholds tied to what depends on the clock

Signal Warn Page Why this number
Fleet spread (payments role) 20 ms 100 ms Above 100 ms, short lock leases stop having meaningful safety margin
Single-host offset 50 ms 500 ms A host this far out is misconfigured, not merely drifting
chrony step events outside boot — any A step breaks monotonicity, which every TTL assumes
Stratum 4 6 or unreachable High stratum means the source chain is degraded
Time since last successful sync 15 min 1 hour Free-running drift is roughly 1–50 ppm, or up to 4 s per day 
groups:
  - name: clock
    rules:
      - alert: FleetClockSpreadHigh
        expr: |
          (max by (role) (node_clock_offset_seconds)
           - min by (role) (node_clock_offset_seconds)) > 0.1
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Clock spread {{ $value | humanizeDuration }} in role {{ $labels.role }}"
          runbook: "Lock leases and off-store TTLs are no longer safely padded."

      - alert: ClockStepped
        expr: increase(node_timex_sync_status[5m]) < 0
        labels: { severity: page }
        annotations:
          summary: "System clock stepped on {{ $labels.instance }} outside boot"

Step 5 — Derive the padding

Once the number is measured, the padding stops being a guess.

MEASURED_SPREAD_MS = 80        # 99.9th percentile over 30 days, payments role
SAFETY_FACTOR = 3

def padded(intended_ms: int) -> int:
    """Only for expiry evaluated OFF the authoritative store. Server-side
    expiry (Redis PX, SQL now() + interval) needs no padding at all."""
    return intended_ms + MEASURED_SPREAD_MS * SAFETY_FACTOR
Timer Intended Padding at 80 ms spread ×3 Padding as a share
Idempotency key TTL 24 h +240 ms 0.0003% — negligible
Lock lease 30 s +240 ms 0.8% — fine
Short lease 2 s +240 ms 12% — uncomfortable
Sub-second lease 500 ms +240 ms 48% — the design is wrong

That last row is the useful output of the whole exercise. A 500 ms lease in a fleet with 80 ms of spread is not a tuning problem; it means the design needs fencing tokens rather than a shorter timer.

Verification and testing

Confirm slewing is actually configured.

chronyc tracking | grep -E 'System time|Update interval'
grep -E '^makestep|^maxslewrate' /etc/chrony/chrony.conf
# expect: makestep limited to the first few updates; maxslewrate present

Confirm the metric appears and is plausible.

curl -s localhost:9100/metrics | grep node_clock_offset_seconds
# expect: a value well under 0.01 on a healthy in-region host

Confirm the alert fires. Inject skew in a staging container and watch the spread cross the threshold.

docker run --rm --cap-add SYS_TIME -d --name skewed alpine sleep 3600
docker exec -u root skewed date -s '+250 milliseconds'
sleep 360
# expect: FleetClockSpreadHigh firing within the 5-minute `for` window

Confirm deduplication survives the injected skew — the reason any of this matters:

./scripts/dedup_suite.sh --skew-ms=-1000
./scripts/dedup_suite.sh --skew-ms=+60000
psql -c "SELECT idempotency_key, count(*) FROM charges
          GROUP BY 1 HAVING count(*) > 1"
# expect: zero rows at every injected offset

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Average offset monitored instead of the spread, so two hosts 40 ms out in opposite directions look healthy Alert on max - min per role. The mean is close to zero by construction and hides exactly the disagreement that matters. max by (role) - min by (role); keep the mean off the dashboard entirely
makestep left unrestricted, and a step during a correction expires live idempotency keys early Set makestep 1.0 3 so steps apply only near boot. Move expiry evaluation to the store where possible so the application clock stops mattering. node_timex_sync_status; a chrony log grep for “System clock was stepped” as a counter
A container image ships without an NTP client and free-runs, drifting seconds per day unnoticed Rely on the host’s clock (containers share the kernel clock by default) and alert on any instance not reporting node_clock_offset_seconds at all. An absent-metric alert: absent(node_clock_offset_seconds{role="payments"})
VM live-migrated or resumed, and the guest clock is minutes behind until the next sync Enable the hypervisor’s guest time-sync agent and force a resync on resume. Detect resume in-process by watching monotonic-versus-wall divergence. clock_resume_detected_total; a step event correlated with a hypervisor migration event
Skew measured once during a project and hard-coded as a constant, then never revisited after a new region was added Re-derive the 99.9th percentile monthly and fail a config check if the code constant is below the measured value. A scheduled job comparing MEASURED_SPREAD_MS in the repo against the live 30-day quantile
What a host with no working NTP does A crystal oscillator drifting at a typical twenty parts per million accumulates about one and a half seconds of error per day. Over a week that is more than ten seconds, which is enough to break a thirty second lease outright. The line is smooth and monotonic, which is exactly why nothing alerts on it unless the offset itself is monitored. offset days without a successful sync → ≈1.5 s per day at 20 ppm smooth, monotonic, and completely silent alert on time-since-last-sync, not just on offset Scope the spread to hosts that interact Two roles measured separately. The payments fleet holds a spread of four milliseconds. The batch fleet drifts to two hundred milliseconds. A single global metric would report two hundred and page the wrong team; per-role metrics report each accurately and only alert where the drift actually threatens a lease or a TTL. payments role spread 4 ms — healthy batch role spread 200 ms One global metric reports 200 ms and pages the wrong team. Scope it, or nobody acts on it.

SRE / observability checklist

  1. node_clock_offset_seconds — Gauge per host. Warn at 50 ms, page at 500 ms; a host this far out is misconfigured.
  2. Fleet spread, max - min by role — The primary metric. Warn at 20 ms, page at 100 ms.
  3. node_timex_sync_status — Gauge. Any transition outside boot means a step occurred and monotonicity was broken.
  4. node_clock_stratum — Gauge. A rising stratum means the upstream source chain is degrading before offsets visibly move.
  5. clock_resume_detected_total — Counter from in-process monotonic-versus-wall divergence. Catches live migrations that NTP metrics miss.
  6. absent(node_clock_offset_seconds) — An absent-metric alert per role, so a host that never reports is not silently assumed healthy.