Traces
Traces show the path of requests through your distributed system. Each trace captures the full journey of a request, including all the services it touched and how long each step took.
When to Use Traces
Traces are ideal for:
- Understanding request flows: See how requests move through services
- Finding performance bottlenecks: Identify slow operations
- Debugging distributed systems: Follow a request across multiple services
- Correlating with logs: Jump from a trace to related log entries
How Traces Work
A trace consists of spans - individual operations within a request. Each span has:
- Service name: Which service performed the operation
- Operation name: What the operation was
- Duration: How long it took
- Parent span: The operation that triggered this one
When instrumented correctly, you can follow a request from the initial HTTP call through all backend services and databases.
Sending Traces
Traces require application instrumentation with OpenTelemetry:
- Add the OpenTelemetry SDK to your application
- Configure the OTLP exporter (see PaaS Full Observability)
- Ensure trace context is propagated between services
Context Propagation
For traces to connect across services, you must propagate trace context. OpenTelemetry handles this automatically for many frameworks, but you may need to configure it explicitly for custom HTTP clients.
See the OpenTelemetry context propagation docs for details.
Viewing Traces
- Go to Grafana
- Open Explore
- Select your Tempo data source
- Search by trace ID, service name, or duration
Finding Traces
You can search for traces by:
- Trace ID: If you have a specific trace ID (e.g., from a log entry)
- Service and operation: Find traces for a particular endpoint
- Duration: Find slow requests
- Status: Find failed requests
Correlating Signals
The power of observability comes from correlating signals:
- From a log entry, extract the trace ID and view the full trace
- From a trace span, jump to related log entries
- From metrics, drill down to traces for slow requests