Role of Jaeger and Tempo

Jaeger and Tempo are distributed tracing backend systems that track and record the entire path and flow of a single user request as it traverses multiple services in an MSA environment.

The concept of a Trace is the entire execution path from the user's initial request to the final response (e.g., the entire product payment request).

The concept of a Span is an individual unit of work that makes up a Trace. Examples include calling an authentication server or executing a DB query. Each span has metadata such as start time, end time, error status, and tags.

Jaeger (pronounced "yay-ger," not "jay-ger" lol) is an open-source tracing system developed by Uber. It indexes collected spans in its own databases like Cassandra or Elasticsearch, demonstrating powerful performance for searching traces by specific tags or execution times.

Tempo, developed by Grafana Labs, is a relatively newer system. Unlike Jaeger, it doesn't create vast indexes; instead, it compresses and stores entire traces in inexpensive object storage. It is optimized for a point-lookup method, precisely pinpointing traces via trace_id extracted from logs or metrics, boasting overwhelming cost-effectiveness and scalability.

Problem Definition #

In modern infrastructure, where environments are split into dozens or hundreds of microservices, unlike traditional server environments, blind spots in visibility arise that cannot be resolved with existing metrics and logs alone.

Fragmented request flow (context loss): When a client calls an API gateway, and the gateway sequentially/parallelly calls services A, B, and C, if an error occurs, it's impossible to immediately identify which service segment failed. Logs left on each server lack a common link, leading to fragmentation.
Inability to identify latency bottlenecks: A metric alarm might indicate that a payment API took 5 seconds and timed out. However, the metric system doesn't tell us that, for example, 1 second was spent in the authentication server, 3 seconds in the inventory server, and 1 second in payment or PG integration. Engineers waste time sifting through logs of each system individually.
Unclear accountability: When a failure occurs, teams might claim their server resources are normal, asking "Isn't it a DB issue?" This leads to each team looking only at their own dashboards, potentially creating communication bottlenecks.

Example #

Let's first consider a normal scenario: in a monolithic environment, a user clicks an order button. Authentication, inventory check, payment, and DB storage all happen within a single WAS Tomcat server. Even if a failure occurs, opening just one catalina.out log file reveals the execution flow stack trace.

Consider a scenario causing failures and leading to debugging dead ends (without distributed tracing). During a Black Friday event, a user places an order. In this complex process: API Gateway -> Auth Service -> Order Service -> Inventory Service (Kafka publish) -> DB, the user receives a 500 error.

Gateway log: Timeout wating for Order Service
OrderService log: Connection refused to Inventory Service
InventoryService log: No error (the request itself was queued and not processed)

As in the situation above, logs alone cannot depict the connected flow of a specific user's request, showing which node it passed through and exactly where it failed. If millions of error logs are generated, time will be spent trying to find the service that is the true root cause, potentially missing the golden hour.

Solution #

Apply standard libraries like OpenTelemetry (OTel) to the code of all services to enable instrumentation and propagate traceparent (trace ID) to the next service via HTTP headers, facilitating context propagation. The collected data is then sent to Jaeger and Tempo.

Extracting trace ID from application logs and correlation analysis #

When distributed tracing is applied, the same trace_id is stamped in the logs of all servers. If an error log is found in a production server terminal, extract the corresponding trace_id.

# Order 서비스 로그에서 특정 주문 실패 에러 검색하여 trace_id 확인
$ grep "Order failed" /var/log/order_service.log | tail -n 1

{"timestamp":"2026-03-02T21:35:10Z","level":"ERROR","message":"Order failed for item 99","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}

Query the Jaeger/Tempo API with the extracted trace_id to analyze bottleneck spans.

Using the identified trace_id 4bf92f3577b34da6a3ce929d0e0e4736, query the Jaeger API to immediately determine which service consumed the most time or threw an error.

Let's use the jq tool to filter only bottleneck segments that took longer than 1 second (1,000,000 microseconds).

# Jaeger API를 호출하여 해당 Trace ID를 구성하는 스팬 중 소요시간(duration)이 1초 이상인 것만 추출
$ curl -s "http://jaeger-query:16686/api/traces/4bf92f3577b34da6a3ce929d0e0e4736" | jq '.data[0].spans[] | select(.duration > 1000000) | {service: .processID, operation: .operationName, duration_ms: (.duration / 1000)}'

{
  "service": "p3",
  "operation": "HTTP GET /api/v1/inventory/check",
  "duration_ms": 5012.5
}
{
  "service": "p5",
  "operation": "SELECT * FROM inventory WHERE item_id = ?",
  "duration_ms": 4980.1
}

With a single terminal result, we can identify that the root cause of the 5-second (5012ms) latency was a database select query (p5) called by p3 in the inventory service.

This allows us to predict issues like locks or slow queries and guide us toward improving them.

In this way, Jaeger and Tempo eliminate vague assumptions and inter-departmental blame, providing data-driven insights into exactly which code or query is degrading the overall system performance.

They play a crucial role in providing clear evidence through flame graph visualizations and JSON data.