Skip to main content
Gå til innhold

Traces

Traces capture the timeline of a single request as it moves through your system, with one span per operation. For an introduction to traces and when to use them, see the OpenTelemetry tutorial and Choosing your signals.

On Platon, traces are stored in Tempo, and you read them in your team's Grafana organization.

Sending traces​

Traces require application instrumentation with OpenTelemetry. If you have not set that up yet:

  1. Read the OpenTelemetry tutorial.
  2. Follow PaaS Full Observability for the environment variables.
  3. See the example apps for a working reference.

Tempo uses the service.name resource attribute to group all spans from one service. Set it once with OTEL_SERVICE_NAME, and every span your application emits inherits it.

Context propagation​

Tracing is most valuable when it crosses service boundaries, for example when one trace captures the enrollment request as it hits enrollment-service, which calls eligibility-checker, which calls course-catalog. For this to work, trace context must propagate between services.

In most cases, adding the SDK and enabling auto-instrumentation is enough, and you do not write propagation code yourself. OTel auto-instrumentation sets and reads the W3C traceparent HTTP header in virtually every mainstream HTTP client and server, and in several message-queue libraries.

You may need to handle propagation manually in these cases:

  • Custom HTTP clients that do not use an auto-instrumented library.
  • Message queues your OTel distribution does not instrument (rare today, since Kafka, RabbitMQ and NATS are all covered).
  • Background jobs with no incoming request (the job is the root span and there is no header to read).

For the mechanics, see the upstream OpenTelemetry context propagation docs.

Put context on your spans​

Spans carry the context you filter and group traces by, and logs carry the account of what happened.

  • Put plenty of identifiers, codes and outcomes on spans, where high cardinality is fine. Tempo stores attributes with each span, whereas Mimir creates a time series for every label combination. A span attribute becomes a label on the span metrics Tempo derives from traces only when Platon adds it to those metrics.
  • Use a general span name, such as GET /courses/{id}. The span name becomes a label on span metrics, so a name that contains an ID, such as GET /courses/42, creates a time series per value. Put the ID in an attribute instead, as the OpenTelemetry guidance on span names advises.
  • Sentences and error detail belong in logs. The account of what happened is easier to read as log lines than as span attributes, and Logs for this span shows those lines next to the trace.
  • Keep attribute values short, because spans have attribute size limits and Tempo query costs scale with span size. A short ID is fine, and an entire request body is too large; see Choosing your signals — Common anti-patterns.

Viewing traces in Grafana​

  1. Go to Grafana.
  2. Open Explore.
  3. Select your Tempo data source.
  4. Search by trace ID, service and operation, duration or status.

Finding traces​

  • Search by trace ID, which you typically take from a log line. Log lines written while a span is active include trace_id when OTel logging is wired up.
  • Search by service and operation to narrow the results to one endpoint on one service.
  • Search by duration to find the slowest requests.
  • Search by status to filter to failures.

Jumping between signals​

In Tempo, a span has a Logs for this span button, plus Request rate and p99 latency buttons where Platon has set them up. In the other direction, View trace on a log line and Query with Tempo on an exemplar open a trace. Choosing your signals — Correlating the three signals shows each link with a screenshot, and When a link is missing covers the usual causes when one doesn't show up.

Query language​

Tempo uses TraceQL. For most teams, the search inputs in the Grafana Explore UI are sufficient, and you can write TraceQL when you need more. See the TraceQL documentation.

Traces are retained for 180 days by default. See Tenants — Data retention for details.

Learn more​