Instrumenting Idempotency Flows with OpenTelemetry

Part of: Distributed Tracing for Deduplication

This is a copy-pasteable runbook for adding OpenTelemetry spans around a dedup check, setting the three standard attributes, exporting to a collector, and confirming the spans land in your tracing backend. Each step is independently verifiable before you move to the next.

Problem statement and prerequisites

What you are implementing: a dedup.check child span around every read/write to your idempotency store, carrying idempotency.key_hash, dedup.result, and lock.fencing_token attributes, exported over OTLP to a collector.

Prerequisites:

• You understand the span model for a dedup decision — specifically why the idempotency key must be hashed before it becomes a span attribute, and why dedup.result needs three states rather than a boolean.
• You have a working dedup check already backed by Redis SET NX or an equivalent conditional write.
• You have an OpenTelemetry Collector reachable from your service (locally via Docker, or a managed collector endpoint).

---

Step-by-step implementation

Step 1 — Install the OTel SDK and OTLP exporter

Python

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

Node.js

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-grpc

Step 2 — Bootstrap the tracer provider

Python

# tracing.py — call init_tracing() once at process startup
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def init_tracing(service_name: str, collector_endpoint: str = "localhost:4317"):
    provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
    exporter = OTLPSpanExporter(endpoint=collector_endpoint, insecure=True)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    return trace.get_tracer(service_name)

Node.js

// tracing.js — require this before any other application code
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
  serviceName: 'payments-service',
  traceExporter: new OTLPTraceExporter({ url: 'localhost:4317' }),
});
sdk.start();

Step 3 — Wrap the dedup check in a child span

Python

import hashlib
from opentelemetry import trace

tracer = trace.get_tracer("payments-service")

def check_idempotency(raw_key: str, store):
    key_hash = hashlib.sha256(raw_key.encode()).hexdigest()[:16]
    with tracer.start_as_current_span("dedup.check") as span:
        span.set_attribute("idempotency.key_hash", key_hash)
        record = store.get(raw_key)
        if record is None:
            span.set_attribute("dedup.result", "miss")
            return None
        current_token = store.current_fencing_token(raw_key)
        if record.fencing_token < current_token:
            span.set_attribute("dedup.result", "conflict")
            span.set_attribute("lock.fencing_token", record.fencing_token)
            raise StaleFencingTokenError(raw_key)
        span.set_attribute("dedup.result", "hit")
        span.set_attribute("lock.fencing_token", record.fencing_token)
        return record.cached_response

Node.js

const { trace } = require('@opentelemetry/api');
const crypto = require('crypto');
const tracer = trace.getTracer('payments-service');

async function checkIdempotency(rawKey, store) {
  const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex').slice(0, 16);

  return tracer.startActiveSpan('dedup.check', async (span) => {
    try {
      span.setAttribute('idempotency.key_hash', keyHash);
      const record = await store.get(rawKey);
      if (!record) {
        span.setAttribute('dedup.result', 'miss');
        return null;
      }
      const currentToken = await store.currentFencingToken(rawKey);
      if (record.fencingToken < currentToken) {
        span.setAttribute('dedup.result', 'conflict');
        span.setAttribute('lock.fencing_token', record.fencingToken);
        throw new Error('StaleFencingTokenError');
      }
      span.setAttribute('dedup.result', 'hit');
      span.setAttribute('lock.fencing_token', record.fencingToken);
      return record.cachedResponse;
    } finally {
      span.end();
    }
  });
}

The diagram below shows how the three-way dedup.result outcome maps to what happens next in the span tree — this is the branch logic every implementation of Step 3 must reproduce, in any language.

Step 4 — Export spans to an OpenTelemetry Collector

Run a local collector for development and testing:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

# Run collector + Jaeger locally with Docker
docker network create otel-net
docker run -d --name jaeger --network otel-net -p 16686:16686 -p 14250:14250 jaegertracing/all-in-one:1.57
docker run -d --name otel-collector --network otel-net -p 4317:4317 \
  -v "$(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:0.102.0 --config /etc/otelcol/config.yaml

Step 5 — Verify in Jaeger

# Trigger a request that produces a dedup miss, then a retry that produces a hit
curl -X POST http://localhost:8080/payments \
  -H "Idempotency-Key: 7c1e-order-8841" -d '{"amount":4200}'
curl -X POST http://localhost:8080/payments \
  -H "Idempotency-Key: 7c1e-order-8841" -d '{"amount":4200}'

# Query Jaeger's HTTP API directly for both dedup.result values
curl -s 'http://localhost:16686/api/traces?service=payments-service&tags=%7B%22dedup.result%22%3A%22miss%22%7D' | jq '.data | length'
curl -s 'http://localhost:16686/api/traces?service=payments-service&tags=%7B%22dedup.result%22%3A%22hit%22%7D' | jq '.data | length'
# Both should return 1 or more — confirming both branches produced a span with the attribute set

Open http://localhost:16686, select payments-service, and search by tag dedup.result=hit. You should see the second request’s trace with a short dedup.check span and no process_payment child — exactly the asymmetry described in the trace diagram on the parent page.

Common gotcha: context loss across async boundaries

The single most common reason a dedup.check span shows up in the backend detached from its parent request span — with no trace_id linking it back — is that the active span context did not survive an async boundary. This is a real failure mode in both languages, not a hypothetical one:

Python. If the dedup check is scheduled onto a thread pool executor (loop.run_in_executor) or a separate asyncio.create_task without explicitly propagating context, the OTel context — which is stored in a contextvars.ContextVar — does not automatically cross into the new thread or task unless you copy it first:

import contextvars
import asyncio

def run_dedup_check_in_executor(loop, raw_key, store):
    ctx = contextvars.copy_context()
    return loop.run_in_executor(None, lambda: ctx.run(check_idempotency, raw_key, store))

Node.js. Callback-based Redis clients or any code path that escapes the AsyncLocalStorage-based context propagation (raw setTimeout, unbound EventEmitter callbacks, or older callback-style database drivers) will produce a dedup.check span with no parent. Prefer promise-based client APIs, which the OTel Node SDK’s context manager tracks correctly, over callback-style APIs, which it does not.

In both cases, the symptom in your tracing backend is identical: a dedup.check span exists, its attributes are correct, but it appears as its own root span — or under the wrong parent — instead of nested under the request span that triggered it. If you see orphaned dedup.check spans during verification, check for a thread pool, task, or callback boundary between the HTTP handler and the dedup check before assuming the instrumentation code itself is wrong.

---

Verification and testing

• Confirm the attribute is present on every span, not just some: curl -s 'http://localhost:16686/api/traces?service=payments-service&limit=20' | jq '.data[].spans[] | select(.operationName=="dedup.check") | .tags[] | select(.key=="idempotency.key_hash")' — every dedup.check span in the sample must return a match.
• Confirm the collector is not silently dropping spans by checking its own metrics endpoint: curl -s http://localhost:8888/metrics | grep otelcol_exporter_send_failed_spans — this should stay at 0.
• Load test the dedup path with a short burst of concurrent identical keys and confirm exactly one trace shows dedup.result=miss and the rest show hit or conflict, never more than one miss for the same idempotency.key_hash within the TTL window.

---

Failure scenarios and debugging

---

SRE / observability checklist

1. dedup.check span count matches request count — instrument a ratio check (dedup_check_spans / total_requests) and alert if it drops below 0.98, indicating some code path bypasses the span.
2. dedup.result cardinality stays at exactly 3 values — hit, miss, conflict. Alert on any other value appearing, which indicates a code defect or an unhandled branch.
3. otelcol_exporter_send_failed_spans — Collector-side counter; alert if greater than 0 over a 5-minute window, since failed exports mean traces silently vanish.
4. idempotency.key_hash present on 100% of dedup.check spans — validate via a collector-side processor and emit dedup_span_missing_key_hash_total; alert on any non-zero rate.
5. Trace-to-log correlation — every structured log line emitted inside check_idempotency must include the active trace_id and span_id so incident responders can pivot from a log line straight into the trace.
6. Sampling retention for conflict traces — confirm via the tail-sampling policy that conflict-tagged traces are retained near 100%, not subject to the baseline probabilistic rate.

---

Related

• Distributed Tracing for Deduplication — parent page covering the span model, W3C trace context vs idempotency key, and sampling strategy in full
• Propagating Idempotency Keys Through Trace Context — carrying the key across service hops via OTel baggage once it leaves this service
• Using Redis SET NX for Distributed Request Deduplication — the storage layer this runbook’s store.get() calls wrap
• Lock Timeout & Lease Management — where the lock.fencing_token value attached to these spans originates
• Monitoring Idempotency Metrics and Dashboards — the aggregate dashboard that should trigger a dive into these traces