OpenTelemetry (OTeL)

In the fast-paced world of software development, ensuring the reliability and performance of distributed systems is significant. Telemetry tools play a crucial role in providing insights into the behavior and health of these systems. However, as technology evolves and architectures become more complex, the need for standardized observability solutions becomes increasingly apparent.

OpenTelemetry is an open-source project designed to revolutionize the way developers approach observability in modern software applications. This post will explore the potential impact of OpenTelemetry on telemetry tools and the future of observability, answering below questions:

• What is OpenTelemetry?
• What is the purpose of OpenTelemetry?
• What doesn’t OpenTelemetry do?

There are a lot of observability tools in the market. They are mature and almost standardized if it is considered as a single isolated item(logging, tracing, metrics). OpenTelemetry treats telemetry data in a more collective approach than other tools.

What is OpenTelemetry?

OpenTelemetry is a collection of APIs, SDKs, and tools. It can be used to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help analyze software’s performance and behavior*.

The significance of OpenTelemetry lies in its ability to standardize observability practices across different systems and programming languages. By offering a unified solution for instrumenting applications, OpenTelemetry simplifies the process of capturing and shipping telemetry data, regardless of the underlying infrastructure, or technology stack.

OpenTelemetry is the second biggest CNCF project after kubernetes. It was created by the merge of OpenTracing and OpenCensus. Scope of the project is creation and collection of all signals (logs, traces, metrics, baggage). The project not only defines transportation protocols and specifications, but also has some implementations and SDKs for different languages.

Key Benefits of OpenTelemetry

• Standardization: OpenTelemetry promotes standardization by providing a common set of APIs and protocols for delivery of the telemetry data. This ensures consistency and interoperability across different tools and platforms. Below specifications and protocols are standardized:
• OTel (OpenTelemetry Architecture)
• OTLP (OpenTelemetry Protocol over HTTP/GRPC)
• OpAMP (Open Agent Management Protocol )
• Semantic Conventions
• Interoperability: OpenTelemetry is vendor-neutral, allowing users to choose the telemetry tools and services that best fit their needs without being locked into a specific vendor’s ecosystem. Each vendor has own implemention for specific additional features but it doesn’t mean that there is a vendor lock or losing flexibity of the OTeL. Each OTeL tool from different vendors can interact with each other.
• Community-driven: OpenTelemetry benefits from ongoing improvements, optimizations, and innovations through a community of individual developers and vendor supported contributors to ensure that OpenTelemetry remains relevant and responsive to the evolving needs of the industry. It is graduating from CNCF projects.

Will OpenTelemetry replace current telemetry tools?

While OpenTelemetry offers significant advantages such as standardization and interoperability. The transition may not be immediate or universal, and organizations may choose to adopt OpenTelemetry incrementally, integrating it alongside their existing tools. Each vendor provides tools which are compatiple with OTeL. Enterprise support can be taken by the current vendor for OTeL for the new tools which are compatible with OpenTelemetry specs.

OTeL has variety of flexible patterns for integration. You use your legacy tools while migrating to OTeL. Legacy tools can remain in use while migrating to OteL. It is not mandotary to replace all the tools in this transition.

OpenTelemetry is gaining momentum in the industry, due to its active development, support from major organizations, and alignment with cloud-native best practices. Over time, it may become the de facto standard for observability in modern software development, leading to a gradual migration away from legacy telemetry tools.

For example, in ELK (Elasticsearch, Logshash, Kibana) or EFK (Elasticseach, Fluentd, Kibana) + Elastic Beats (metricbeats, auditbeats, healthbeats…) stack, logstash and beats probably will be replaced by OpenTelemetry collector in the first step. Fluentd can remain depending on the architechtural decisions. However, elasticsearch and kibana will not be replaced by OTeL in the future because these are responsible for storing and visualizing the data which are not covered by OpenTelemetry.

https://www.elastic.co/guide/en/observability/current/open-telemetry.html

What does not OpenTelemetry do?

OpenTelementry does not provide any applications to store or visualize the data. Any OTeL tool can send telemetry data to collector and collector can route it to the backend. Even though OpenTelemetry collector is a vendor agnostic tool, vendors can provide their own collector. It is like kubernetes implementation. Each vendor has own collector at the end of the day, but each of them can interoperate with each other.

OpenTelemetry is not only a blue print or guide. There are some implementations in the repository.

Implementations

• Collector
• SDKs & Language APIs

Collector

OpenTelemetry Collector is the cornerstone of the project. It contains all standardized tools and plays a crucial role in OpenTelemetry architecture. There are two modes:

• Gateway (Forward signals)
• Agent (Collect signals)

These modes can be available together into single collector. Gateway functionality, which looks like reverse proxy combined with simple data pipeline, is simpler. It receives data in OTeL protocols, processes and then forwards them to another point into OTeL protocol. These gateways are stateless. It can be under a loadbalancer as well.

Agent mode is generating or receiving signals. For example, scraping log files, collecting host metrics can generate specific signals. It is open for modular extension and more vendor specific features can be added in this mode.

There are two open source repositories for this application:

• opentelemetry-collector-core: This is the base lib of a generic collector. It can be extended by any vendor. It provides basic functionalities. Basic installable distribution is otelcol .
• opentelemetry-collector-conrib: This is the community distribution of the collector. It contains all vendor natural components. This one can be installed as a otelcol-contrib.

For example, splunk, redhat, azure and appdynamics provide its own collectors and additional features. This specific implementation needs to be installed to collect signals, but these signals can be forwarded to open source implementation.

SDKs & Language APIs

For each language and framework, there is an instrumentation to export required signals. Maturity of the tools can be followed for specific languages and signals. For Java and C#/.NET, all tools are stable.

Conclusion

In summary, OpenTelemetry has the potential to reshape the telemetry landscape by providing a standardized and interoperable solution for observability in distributed systems. While the transition may take time, OpenTelemetry represents a significant step forward in empowering developers and operators to observe effectively not only their applications, but also their entire system.

As technology continues to evolve, OpenTelemetry will play a very important role in ensuring the reliability, performance, and resilience of modern software systems.