OpenTelemetry Collector Pipelines: Building Production Observability Stacks

OpenTelemetry Collector Pipelines for Production Observability

OpenTelemetry Collector pipelines are the backbone of modern observability infrastructure. The Collector acts as a vendor-agnostic proxy that receives, processes, and exports telemetry data — traces, metrics, and logs — from your applications to any backend. Instead of coupling your services to specific monitoring vendors, you route everything through the Collector and swap backends without code changes. That decoupling is the single most valuable property the Collector buys you: the day your vendor doubles its prices, you change an exporter block rather than redeploying every service.

This guide covers building production-grade configurations, from simple single-node setups to multi-tier architectures handling millions of spans per second. You will learn how to filter noise, enrich data, manage sampling, and reduce observability costs without losing visibility. Along the way we will look at the failure modes that bite teams in production — memory blowups, dropped spans, and sampling decisions made before the trace is complete.

Understanding Collector Architecture

The Collector has three main component types: receivers (data ingestion), processors (data transformation), and exporters (data output). These components chain together into pipelines, with each pipeline handling one signal type: traces, metrics, or logs. A single Collector process can run many pipelines at once, so one binary often ingests OTLP traces, scrapes Prometheus metrics, and tails container logs simultaneously.

# Basic collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true

processors:
  batch:
    send_batch_size: 8192
    timeout: 200ms
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
  resourcedetection:
    detectors: [env, system, docker, gcp, aws, azure]

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Core vs Contrib: Choosing the Right Distribution

One decision trips up newcomers immediately: there are two Collector distributions. The otelcol core distribution ships only stable, widely-used components, whereas otelcol-contrib bundles hundreds of community components like the k8sattributes, tail_sampling, and loki exporter used throughout this guide. Most production deployments need contrib, but pulling in every component also widens your attack surface and image size.

For tighter control, teams increasingly build a custom distribution with the OpenTelemetry Collector Builder (ocb), which compiles a binary containing only the receivers, processors, and exporters you actually use. The result is a smaller image, faster startup, and a dependency tree you can fully audit. A common pattern is to prototype with contrib, identify the exact component set, then lock it down with a builder manifest before going to production.

Advanced Processor Pipelines

Raw telemetry data is noisy and expensive to store. Processors let you filter, sample, transform, and enrich data before it reaches your backend. Moreover, the right processor chain can reduce storage costs by an order of magnitude while keeping the signals that matter — benchmarks from teams running high-traffic services commonly report 70 to 90 percent volume reductions once health-check spans and successful, fast requests are sampled away.

# Advanced processor configuration
processors:
  # Tail-based sampling — keep errors + slow traces + sample rest
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces
        type: latency
        latency: {threshold_ms: 2000}
      - name: probabilistic-sample
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

  # Attribute processing — redact PII, add context
  attributes:
    actions:
      - key: user.email
        action: hash
      - key: http.request.header.authorization
        action: delete
      - key: deployment.environment
        value: production
        action: upsert

  # Span filtering — drop health checks
  filter/traces:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        - 'name == "SELECT 1"'

  # Metric transformation — aggregate and rename
  transform/metrics:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["service.namespace"], "production")
      - context: metric
        statements:
          - set(description, "") where name == "process.runtime.jvm.memory.usage"

  # Log body parsing — extract structured fields
  transform/logs:
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, ParseJSON(body), "insert") where IsMatch(body, "^\\{")
          - set(severity_text, "ERROR") where IsMatch(body, "(?i)error|exception|fatal")

Additionally, processor ordering matters significantly. Always place the memory_limiter first to prevent out-of-memory crashes, followed by filtering processors to reduce volume early, then enrichment processors, and finally batching. Ordering is not cosmetic — the Collector runs processors in the exact sequence listed, so batching before filtering wastes work assembling batches you immediately discard.

Head Sampling vs Tail Sampling: Getting the Trade-off Right

Sampling is where most cost-versus-fidelity decisions live, and the two strategies behave very differently. Head sampling decides at the start of a trace, in the SDK, before any spans exist — it is cheap and stateless but blind, so it will happily drop the one request that later threw a 500. Tail sampling, configured above, waits until the full trace arrives so it can keep every error and every slow request while sampling the boring successes.

The catch is that tail sampling is stateful and memory-hungry: the Collector must buffer all spans of a trace until decision_wait elapses. That requirement has a sharp consequence in distributed deployments. If two Collectors each see half of a trace, neither can make a correct decision. This is exactly why the gateway tier below uses a groupbytrace processor and consistent routing — all spans for a given trace ID must land on the same Collector instance. A pragmatic middle ground is light head sampling in the SDK to cap raw volume, followed by tail sampling at the gateway for intelligent retention.

Multi-Tier Collector Architecture

For production deployments handling significant volume, use a two-tier architecture. Agent Collectors run as sidecars or daemonsets, collecting and forwarding data with minimal processing. Gateway Collectors run as a central, horizontally-scaled service, performing heavy processing like tail sampling that requires cross-service context. The agent tier stays cheap and stateless; the gateway tier carries the stateful, memory-intensive work.

# Gateway Collector — central processing tier
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4096

  # Group spans by trace ID for tail sampling
  groupbytrace:
    wait_duration: 15s
    num_traces: 500000
    num_workers: 4

  tail_sampling:
    decision_wait: 20s
    num_traces: 500000
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-high-latency
        type: latency
        latency: {threshold_ms: 1500}
      - name: sample-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

  batch:
    send_batch_size: 16384
    timeout: 500ms

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, groupbytrace, tail_sampling, batch]
      exporters: [otlp/tempo]

Notice the sending_queue and retry_on_failure blocks on the exporter. These are not optional niceties at scale. When the downstream backend hiccups, the persistent queue absorbs the backlog and retries with backoff instead of dropping data on the floor. For routing all spans of a trace to the same gateway instance, teams typically place a load balancing exporter in front of the gateway pool, keyed on trace ID.

Kubernetes Deployment with Helm

# values.yaml for opentelemetry-collector Helm chart
mode: daemonset

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
    filelog:
      include:
        - /var/log/pods/*/*/*.log
      exclude:
        - /var/log/pods/*/otc-container/*.log
      operators:
        - type: router
          routes:
            - output: parser-docker
              expr: 'body matches "^\\{"'
        - id: parser-docker
          type: json_parser
          timestamp:
            parse_from: attributes.time
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'

  processors:
    k8sattributes:
      extract:
        metadata:
          - k8s.pod.name
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.node.name
      pod_association:
        - sources:
            - from: resource_attribute
              name: k8s.pod.ip

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

The k8sattributes processor is what turns raw spans into something a human can debug. By correlating each pod IP with the Kubernetes API, it stamps every span and log with the pod name, namespace, deployment, and node. When an alert fires, that enrichment is the difference between “a span errored somewhere” and “the checkout deployment in payments on node ip-10-0-3-12 errored.” Run it in the agent tier, where the pod IP is still locally known.

Monitoring the Collector Itself

An observability pipeline that silently drops data is worse than no pipeline, because it manufactures false confidence. The Collector exposes its own internal metrics on the service.telemetry endpoint, and a handful of them deserve standing alerts. Watch otelcol_processor_refused_spans and otelcol_processor_dropped_spans for back-pressure from the memory limiter, otelcol_exporter_send_failed_spans for backend trouble, and the exporter queue size approaching its configured capacity.

A practical rule is to alert when refused or dropped counts climb above zero for more than a few minutes, since under healthy conditions they should sit flat at zero. Pair this with a dashboard tracking memory against the limit_mib ceiling. For teams already standardizing on Prometheus and Grafana, the patterns in the Kubernetes monitoring with Prometheus guide drop straight onto these self-telemetry metrics.

When NOT to Use OpenTelemetry Collector

The Collector adds network hops and operational complexity. If your organization uses a single observability vendor and has fewer than 10 services, sending telemetry directly from SDKs to the backend is simpler and one less thing to operate. Furthermore, if you are in a regulated environment where data must not pass through intermediary services, direct export may be required for compliance, and inserting a Collector creates an extra place where sensitive data is buffered.

Consequently, small teams running simple architectures should start with direct SDK exporters and introduce the Collector when they need vendor-agnostic routing, data transformation, or cost optimization through sampling. The Collector shines at scale but adds overhead for simple setups. A reasonable trigger point is the moment you find yourself wanting to change a sampling rule or redact a field without redeploying applications — that is precisely the friction the Collector removes.

Key Takeaways

A well-built Collector deployment provides the flexibility and power needed for production observability at scale. Start with a simple configuration, add processors as your needs grow, and consider a gateway tier when volume demands tail sampling. The vendor-agnostic approach protects your investment regardless of which backends you choose today or switch to tomorrow.

Key Takeaways

• Start with a solid foundation and build incrementally based on your requirements
• Test thoroughly in staging before deploying to production environments
• Monitor performance metrics and iterate based on real-world data
• Follow security best practices and keep dependencies up to date
• Document architectural decisions for future team members

For related DevOps topics, explore our guide on Kubernetes monitoring with Prometheus and Docker container best practices. Additionally, the official OpenTelemetry Collector documentation and contrib repository are essential references.

In conclusion, OpenTelemetry Collector pipelines are an essential topic for modern software development. By applying the patterns and practices covered in this guide, you can build more robust, scalable, and maintainable systems. Start with the fundamentals, iterate on your implementation, and continuously measure results to ensure you are getting the most value from these approaches.