Skip to main content

📡 OpenTelemetry Collector

The OpenTelemetry Collector is the absolute backbone of our tracing infrastructure! It gathers, processes, and exports all telemetry data.

In our stack, it serves as the dedicated, high-speed bridge between your applications and our tracing backend, Grafana Tempo. 🌉

otel architecture

🏗️ Two-Tier Tracing Architecture​

To easily handle massive scale and ensure our traces never break, we deploy a slick two-tier collector setup:

  • 🔀 External Collector (Router): All applications blast their traces here! Because this layer scales up and down rapidly under load, individual spans belonging to the same request trace might hit different pods. To solve this, the external collector acts as a smart router—ensuring all spans of the same trace are sent to the same internal pod.
  • 🧠 Internal Collector (Processor): Only the external collector talks to this layer. Since the spans were perfectly routed, the internal collector can effortlessly reassemble the complete trace hierarchy! It also acts as our bouncer—filtering out noisy health, liveness, and metrics traces before securely exporting the clean data to Tempo. 🧹

This dual-layer approach gives us extreme high availability while keeping storage costs incredibly low! 🚀