How to add OpenTelemetry tracing to a deployed app

Logs tell you what happened. Traces tell you where the time went. When a request takes three seconds and every individual log line looks fine, tracing is the tool that shows you the 2.6 seconds spent in a database call you'd forgotten was in that code path.

OpenTelemetry is the vendor-neutral way to produce that data, and its reputation for being complicated comes almost entirely from the configuration surface rather than from the setup, which is genuinely short. This is the short version — get traces flowing first, tune afterwards.

The three pieces

• Instrumentation — code that creates spans. Auto-instrumentation wraps your HTTP server, database driver and HTTP client without you touching anything.
• An exporter — ships spans somewhere over OTLP. This is a handful of environment variables.
• A backend — receives and displays them: Grafana Tempo, Jaeger, Honeycomb, Datadog, Signoz, or anything else speaking OTLP.

You need all three, and the most common reason a first attempt shows nothing is that one of them was skipped — usually the exporter endpoint, which defaults to localhost.

Node: auto-instrumentation in one file

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

// tracing.js — must be loaded BEFORE anything else
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),   // reads OTEL_EXPORTER_OTLP_ENDPOINT
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

process.on("SIGTERM", () => {
  sdk.shutdown().finally(() => process.exit(0));
});

Load order is the whole game. Auto-instrumentation works by patching modules as they're required, so it has to run before your application imports Express or `pg`. Use the `--require` flag rather than importing it from your app:

{
  "scripts": {
    "start": "node --require ./tracing.js dist/server.js"
  }
}

Python: the zero-code path

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install   # installs matching instrumentation libs

Then prefix your start command. No import, no initialisation file — the wrapper sets everything up before your application loads:

opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port $PORT

# Django, Flask, Celery — same wrapper
opentelemetry-instrument gunicorn -b 0.0.0.0:$PORT app:app

One gotcha with Gunicorn and Uvicorn workers: instrumentation initialises per worker process. That's usually correct, but if you see duplicate or missing spans under a pre-fork server, check whether you're initialising in the parent and forking after.

The environment variables that matter

OpenTelemetry is configured almost entirely by environment variables, which is convenient on a platform where you set those per app. Four do most of the work:

OTEL_SERVICE_NAME=checkout-api
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.your-backend.example.com
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=<token>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.4.2

Set them on the app rather than committing them, since the headers value is a credential:

curl -X PATCH https://api.pandastack.ai/v1/apps/$APP_ID \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"env":{
        "OTEL_SERVICE_NAME":"checkout-api",
        "OTEL_EXPORTER_OTLP_ENDPOINT":"https://otlp.your-backend.example.com",
        "OTEL_EXPORTER_OTLP_HEADERS":"x-api-key=<token>",
        "OTEL_RESOURCE_ATTRIBUTES":"deployment.environment=production"
      }}'

`OTEL_SERVICE_NAME` is the one people forget, and the symptom is memorable: every service in your trace view is called `unknown_service`, and a distributed trace becomes unreadable.

Add spans where auto-instrumentation can't see

Auto-instrumentation covers the boundaries — HTTP in, HTTP out, database queries. It knows nothing about your own logic, so the expensive loop in the middle of a request shows up as unexplained time between two spans. Add spans there:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def score_candidates(candidates, model_version):
    with tracer.start_as_current_span("score_candidates") as span:
        span.set_attribute("candidate.count", len(candidates))
        span.set_attribute("model.version", model_version)
        results = [score(c) for c in candidates]
        span.set_attribute("result.count", len(results))
        return results

Attributes are what turn a trace into an investigation. "This request was slow" is mildly useful; "requests with candidate.count above 500 are slow" is a fix. Put the dimensions you'd want to group by on the span — counts, sizes, versions, cache hit or miss, tenant identifiers if your platform allows them.

Sampling, before your bill notices

Tracing every request is fine in staging and expensive in production — most backends bill per span ingested. The default sampler records everything, so this is a decision you make deliberately or have made for you by an invoice.

# Sample 10% of traces, head-based
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

`parentbased_traceidratio` is the important part of that: it makes the sampling decision once at the start of a trace and propagates it, so a sampled trace is complete across every service rather than a scattering of disconnected spans.

Head-based sampling has one real weakness — it decides before knowing whether the request was interesting, so a 10% sample throws away 90% of your errors too. If your backend supports tail-based sampling through a collector, sampling on "was there an error or was it slow" is strictly better. Start with a ratio, move to tail-based when volume justifies the collector.

My traces are empty

1. Endpoint unset. The default is localhost:4318 — with nothing listening there, spans are exported into the void and the SDK doesn't complain loudly.
2. Instrumentation loaded too late. Node needs --require; Python needs the opentelemetry-instrument wrapper. Importing the setup from inside your app is usually too late.
3. The process exits before the exporter flushes. Short-lived jobs and serverless functions need an explicit shutdown or force-flush, or the last batch is lost.
4. Sampling set to zero somewhere. Check OTEL_TRACES_SAMPLER_ARG in the deployed environment, not in your local .env.
5. Wrong protocol. The OTLP HTTP exporter wants port 4318 and the gRPC one wants 4317; mixing them fails in ways the error message doesn't make obvious.

Turn on the debug exporter locally before blaming the backend. If spans print to your console, the instrumentation works and the problem is the endpoint or its credentials:

OTEL_TRACES_EXPORTER=console opentelemetry-instrument python app.py

Connect traces to logs

The highest-value ten minutes of this whole exercise is putting the trace ID into your log lines. Then a slow trace leads directly to the log lines from that exact request, and a log line with an error leads to the trace showing what the request was doing:

const { trace } = require("@opentelemetry/api");

function logWithTrace(level, msg, fields = {}) {
  const span = trace.getActiveSpan();
  const ctx = span?.spanContext();
  console.log(JSON.stringify({
    level, msg, ...fields,
    trace_id: ctx?.traceId,
    span_id: ctx?.spanId,
  }));
}

That single field is what turns two separate tools into one investigation, and it costs a wrapper function.

Recap

1. Install the SDK and auto-instrumentation; load it before your app, via --require or opentelemetry-instrument.
2. Set OTEL_SERVICE_NAME, the OTLP endpoint, and its auth header as app environment variables.
3. Verify locally with the console exporter before debugging the backend.
4. Add custom spans with attributes around your own expensive logic.
5. Configure parentbased_traceidratio sampling before production volume arrives.
6. Put trace_id into every log line.