estudy 한국어/English

OpenTelemetry Structure

1,458 단어·3 분·원문(.md)

OpenTelemetry, or OTel, is a standard framework for observability essential in cloud-native environments.

It provides a unified way to collect, process, and export traces, metrics, and logs without being dependent on specific vendors like Datadog.

OTel Architecture #

OTel is not just a library for sending data; it has a layered architecture as described below.

API & SDK #

These parts operate within the application code.

  • API: An abstract tool for instrumentation. It contains no actual operational code but defines signals for starting spans. Used by library developers.
  • SDK: The actual implementation of the API. It buffers collected data in memory, performs sampling, and ultimately decides where to send the data.

A Span is the smallest unit of work in OTel. Simply put, it's a record of a single operation that occurred from a specific point to another specific point among the various stages a request goes through from entering to exiting a system.

We'll learn more about spans below.

Data Model (Signals) #

OTel processes three core signals:

  • Traces: Tracks the flow of requests (in units of spans).
  • Metrics: Numerical data such as CPU usage and request counts.
  • Logs: Records of events that occurred at specific points in time (standardization currently in progress).
  • Resources: Metadata about the entity generating the data, such as service name, hostname, and container ID.

OpenTelemetry Collector #

This is an intermediate server that receives data sent from applications, processes it, and then forwards it to storage.

Thanks to this structure, you can change data storage without modifying your application code.

Key Components of a Span #

  • Name: A name describing the operation performed, e.g., GET /api/v1/users, SELECT * from orders.
  • Trace ID: A unique ID identifying the entire transaction; all related spans share the same trace ID.
  • Span ID: A unique ID for that specific span.
  • Parent Span ID: The ID of the parent span that initiated this operation, forming a tree structure.
  • Start & End Timestamp: The exact times when the operation started and ended. The difference between these becomes the latency of the operation.
  • Status: Indicates whether the operation was successful: unset, ok, error.
  • Attributes: Key-value metadata for filtering or analysis, such as http.method: GET, db.system: postgresql.
  • Events: Records of specific occurrences during span execution, such as cache hits or retry starts, including timestamps.
  • Links: Used to define relationships with other traces or spans, for example, connections between development tasks in a batch processing job.

Collector's Internal Pipeline Structure #

The Collector consists of a 3-stage pipeline, typically managed in a config.yaml file.

  1. Receivers: The entry point for receiving data. Supports various formats like OTLP (OTel standard protocol), Jaeger, and Prometheus.
  2. Processor: Processes data, such as masking sensitive information, batching data, or adding specific tags.
  3. Exporters: Sends data to its final destination, such as Prometheus, Jaeger, AWS CloudWatch, Elasticsearch, etc.

Context Propagation #

Understanding context propagation is important; it's a crucial concept in distributed systems.

When service A calls service B, it sends the trace ID in HTTP headers, etc., to link them into a single long trace.

OTel strongly supports auto-instrumentation, which handles this process automatically.

Example #

This is the core configuration file used when running the Collector.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:     # 데이터를 묶어서 보내 효율 향상
  memory_limiter: # 메모리 부족 방지
    check_interval: 1s
    limit_mib: 2000

exporters:
  logging:   # 터미널에 로그 출력 (디버깅용)
    verbosity: normal
  otlp/jaeger:
    endpoint: "jaeger:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging, otlp/jaeger]

There's also a way to extract traces with minimal code changes.

# 관련 패키지 설치
pip install opentelemetry-distro \
    opentelemetry-exporter-otlp

# 계측 설정 자동 설치 (Flask, Requests 등 라이브러리용)
opentelemetry-bootstrap -a install

# 애플리케이션 실행 (Collector로 데이터 전송)
export OTEL_SERVICE_NAME="my-awesome-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

opentelemetry-instrument \
    --traces_exporter otlp \
    --metrics_exporter otlp \
    python app.py

You can also manually instrument within the code when you want to measure the detailed performance of business logic.

from opentelemetry import trace

# 트레이서 생성
tracer = trace.get_tracer(__name__)

def heavy_logic():
    # 'my_span'이라는 이름으로 구간 측정 시작
    with tracer.start_as_current_span("process_payment") as span:
        span.set_attribute("payment.amount", 50000) # 커스텀 메타데이터 추가
        
        # 비즈니스 로직 수행
        print("결제 처리 중...")
        
        # 에러 발생 시 기록 예시
        # span.record_exception(e)
        # span.set_status(trace.Status(trace.StatusCode.ERROR))

./otelcol --config=config.yaml # collector 상태 확인

grpcurl -plaintext localhost:4317 opentelemetry.proto.collector.trace.v1.TraceService/Export ## gRPC 엔드포인트 테스트 

env | grep OTEL # 현재 환경의 OTel 환경변수 확인

In conclusion, OTel focuses on how to collect data, leaving storage and visualization to other tools.

Therefore, when adopting OTel, the next decisions are where to deploy the Collector and which backend (Jaeger, Prometheus) to use for receiving the data.

Hierarchical Structure and Propagation of Spans #

Spans come together to form a single trace.

  • Root span: The first span created when a request enters, having no parent.
  • Child span: Sub-operations that occur within a parent span, such as database calls or API calls.

To maintain this structure, context propagation occurs during inter-service communication.

When service A calls service B, it sends the trace ID and parent span ID in HTTP headers like traceparent, allowing B to know whose child it is.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

# 트레이서 가져오기
tracer = trace.get_tracer(__name__)

def process_order(order_id):
    # 'order_logic'이라는 이름의 Span 시작
    with tracer.start_as_current_span("order_logic") as span:
        # 1. 속성(Attributes) 추가: 검색 및 분석용
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.region", "KR")

        try:
            # 비즈니스 로직 수행
            do_database_work()
            
            # 2. 이벤트(Events) 추가: Span 내 특정 시점 기록
            span.add_event("database_query_finished", {"rows": 42})
            
        except Exception as e:
            # 3. 상태(Status) 및 예외 기록
            span.set_status(Status(StatusCode.ERROR))
            span.record_exception(e)
            raise e

def do_database_work():
    # 중첩된 자식 Span 생성
    with tracer.start_as_current_span("db_query") as child_span:
        child_span.set_attribute("db.statement", "SELECT * FROM orders")
        # DB 작업 수행...

Span Attributes Rules #

OTel predefines attribute names for vendor neutrality. Adhering to these allows tools like Jaeger and Grafana to automatically recognize and visualize the data on dashboards.

http.method, http.status_code, db.system, net.peer.name, service.name

HTTP-related and DB system attributes are understandable (method, code, DB type (MySQL, Redis, PostgreSQL...)).

net.peer.name is the target host of the call, and service.name is literally the service name, e.g., payment-service for a resource.

OTel Data Transmission with curl

When the Collector is running, you can send dummy span data to its HTTP endpoint to verify its proper operation.

# OTLP HTTP receiver (4318 포트)로 간단한 JSON 데이터 전송
curl -X POST http://localhost:4318/v1/traces \
-H "Content-Type: application/json" \
-d @- <<EOF
{
 "resourceSpans": [{
   "resource": {
     "attributes": [{"key": "service.name", "value": {"stringValue": "test-manual-curl"}}]
   },
   "scopeSpans": [{
     "spans": [{
       "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
       "spanId": "00f067aa0ba902b7",
       "name": "manual-span-test",
       "kind": 1,
       "startTimeUnixNano": "$(date +%s)000000000",
       "endTimeUnixNano": "$(($(date +%s) + 1))000000000"
     }]
   }]
 }]
}
EOF

The 4318 server here is an OTel Collector server, which is not a server we built, but rather a software package or a server launched as a Docker image.

Its purpose is to be a multi-purpose data relay server, and the OTel Collector is a standalone executable written in Go.

Considerations for Span Design #

  1. Too many spans: Creating a span every time within a loop can lead to significant overhead. Create them at meaningful business boundaries.
  2. High Cardinality: It's fine to include many unique values like order IDs or user IDs in attributes, but putting such values directly in the span name itself reduces indexing efficiency. Keep names static and use attributes for IDs.
  3. Context Leak: In asynchronous programming, contextvars or similar mechanisms must be properly managed to prevent the parent span's context from being lost.
SRE/question/q_32.md