Let's Code: Writing Observable Code

Enjoying this newsletter? Please share it with your network and encourage them to subscribe to receive more articles on observability.

In this article, we will dive into a bit of code to illustrate how engineers can write good observable code without worrying about the infrastructure of collecting, forwarding, and storing telemetry data. Just as good engineers will write proper readable code, with appropriate tests, great engineers will also judicially output signals so that the behaviors of their code can be observed in a distributed system. The first part of this article will discuss how a developer can use common libraries to output logs, metrics, and traces regardless of how they'll be collected.

I'll illustrate some of these concepts using C# and ASP.NET, but it's worth noting that many languages and frameworks have similar libraries that enable engineers to achieve the same level of observability. I've chosen to use ASP.NET for two reasons. Firstly, it's a framework that I'm intimately familiar with. Secondly, ASP.NET is unique in that it has native APIs for all three types of signals - logs, metrics, and traces. This means that developers can create observable code without needing to rely on additional libraries.

I'll cover other frameworks and languages in future articles, but let's start here with C# and ASP.NET.

There is a fully working code example here: https://github.com/drewby/observable-code-aspnet

ASP.NET Observability APIs

ASP.NET offers several APIs that allow developers to instrument their code with signals like logs, metrics, and traces. These APIs are implemented in the framework libraries but are closely related to the OpenTelemetry API specification.

The first API, ActivitySource, is used to create and manage distributed traces. This allows developers to track request flows across different services, making it easier to identify bottlenecks and issues in complex, distributed systems.

Each span within a trace is represented by an Activity, which provides methods for logging events and attributes. By using Activities, developers can get a more comprehensive view of how their code behaves across different services and identify the root cause of issues more quickly.

The Meter API emits metrics from an application, which allows developers to monitor key performance indicators and detect trends. Metrics can be Counters, Gauges, or Histograms.

The Logger API emits logs from an application, providing methods for logging messages at different severity levels and adding structured data. Logs can capture important events with more details than what would go into the other signals.

Finally, the Baggage API - while not a signal itself - is closely related to Observability and is part of the OpenTelemetry specification. Baggage represents context that is propagated across a trace, allowing developers to correlate events within the trace and diagnose contextual issues. By using baggage, developers can get a more complete picture of how their application behaves in production and diagnose issues more efficiently.

Logging

In ASP.NET Core, the built-in ILogger interface provides methods for logging messages with different severity levels, such as Information, Warning, Error, and Critical.

To use ILogger in an ASP.NET Core application, we can obtain an instance of it through dependency injection, typically in the constructor of a class:

public class ToDoController : ControllerBase
{?
? ? private readonly ILogger<ToDoController> _logger;?

? ? public ToDoController(ILogger<ToDoController> logger)?
? ? {?
? ? ? ? _logger = logger;?
? ? }?
}        

Once we have an instance of ILogger, we can log messages using methods corresponding to the desired severity level. In the example provided, the LogInformation method is used to log an informational message:

_logger.LogInformation("User '{user}' searched for '{searchText}' in the ToDo items database", userName, searchText);        

When instrumenting the application with OpenTelemetry or similar libraries, additional information, such as the traceId and spanId of the current Activity, can be automatically included in the log entries. This allows for better correlation between logs and traces, making it easier to understand the relationship between events and the application's execution flow.

It's important to note that while logging provides valuable information about an application's behavior, it can also impact performance, particularly when logging at a high volume or with a high severity level. Therefore, it's essential to balance the level of detail and frequency of logging with the performance requirements of the application. This can be achieved by configuring log levels, filtering, and output providers in the application's configuration settings.

Metrics

To instrument code with metrics, we can use a Meter in the System.Diagnostics.Metrics namespace to generate metrics for a specific application or library. A Meter provides an interface for creating and updating various types of metric instruments, such as Counters, Gauges, and Histograms. The Meter is identified by a name, which should be the same across all instances of the application. This name helps in processing, filtering, or routing metric data during collection.

Creating a Meter involves instantiating it with the application or library name:

using var meter = new Meter("todoApp");        

It's good practice to create one Meter and reuse it across a process. In ASP.NET, this could be done by creating a static variable or by passing the Meter to controllers and services using dependency injection.

The following are brief descriptions of the different types of metric instruments provided by a Meter.

Counter

A Counter is a metric instrument that represents a value that can only increase or remain constant over time. It's typically used for counting events or measuring the total quantity of something. In the example provided, a Counter is created to count the number of ToDos created by the service instance

var subCallsCounter = meter.CreateCounter<int>("ToDos");        

Later in the request, the Counter is updated with the number of ToDos:

subCallsCounter.Add(call.Calls.Count);        

Gauge

A Gauge is a metric instrument that represents a value that can increase or decrease over time. It's often used to measure instantaneous values, such as the current number of active connections or memory usage. To create a Gauge, use the CreateGauge method on the Meter instance:

var memoryUsageGauge = meter.CreateGauge<double>("memoryUsage");        

To update the Gauge value, use the Set method:

memoryUsageGauge.Set(currentMemoryUsage);        

Histogram

A Histogram is a metric instrument that measures the distribution of values over time. It's useful for tracking the distribution of response times, sizes, or other measurements. To create a Histogram, use the CreateHistogram method on the Meter instance:

var responseTimeHistogram = meter.CreateHistogram<double>("responseTime");        

To update the Histogram with a new value, use the Record method:

responseTimeHistogram.Record(responseTime);        

These metric instruments can be collected by an SDK and sent to a backend analytics system for further analysis and visualization.

Tracing

To instrument code with tracing, we can use an ActivitySource to generate new Activity objects in the current process. The name of the ActivitySource can be the application or library name, and it will be used to process, filter, or route trace data during collection.

To create an ActivitySource, we can use the following code:

var activitySource = new ActivitySource(appInfo.Name);        

It's good practice to create one ActivitySource and reuse it across a process. In ASP.NET, this could be done by creating a static variable or by passing the ActivitySource to controllers and services using dependency injection.

When there's a unit of work that should have a trace span, we can create a new Activity using the ActivitySource:

using var activity = activitySource.StartActivity(
   ActivityKind.Server,  
   name: "Get ToDo Items", 
   tags: new Dictionary<string, object?> 
   { 
      ["user"] = userName, 
      ["searchText"] = searchText 
   });        

This Activity gets an ActivityKind of Server, a name that describes the work being done, and any additional tags that will be useful when reviewing the Activity later.

When the ActivitySource creates the Activity, it will link it to any existing context by assigning a TraceId and parent SpanId from the current context. This context may be in the headers of the incoming request or created elsewhere in the process, but as a developer, we don't need to worry about it.

Likewise, if any subcalls generate new Activity objects, they will be associated as a child of the Activity we created here. Because of the using statement, this Activity will be disposed of once it falls out of scope. Once disposed of, the Activity can be collected by an SDK and sent to a backend analytics system.

Baggage

Baggage is a feature in OpenTelemetry that allows context to be propagated across requests in a distributed system. Baggage is not a signal itself but is closely related to Observability and is part of the OpenTelemetry specification.

In ASP.NET, we can use Baggage to propagate context across HTTP requests. For example, we could use Baggage to propagate a user ID or a search term across requests, allowing us to easily correlate events within the trace and understand the context of each request.

Baggage is available from the Activity we created in the Tracing section. Once we have an Activity context, we can use the Baggage class to get and set Baggage values. Here's an example:

using var activity = activitySource.StartActivity("Get ToDo Items")

// Set a Baggage value
activity.SetBaggageItem("userId", userId.ToString());

// Get a Baggage value
var userId = int.Parse(activity.GetBaggageItem("userId") ?? "0");;        

In this example, we're setting a Baggage value for the user ID and then retrieving it later using the GetBaggageItem method. We could use a similar approach to propagate other contexts across requests, such as a search term or a correlation ID.

It's important to note that Baggage values are propagated across requests using HTTP headers, which can impact performance. Therefore, it's essential to only propagate necessary context and avoid including sensitive or large values in Baggage. Also, be careful that your service doesn't unintentionally forward sensitive data in baggage to external services.

Write Observable Code

Writing observable code is a crucial aspect of building reliable and scalable distributed systems. By providing signals like logs, metrics, and traces, developers can gain visibility into the behavior of their code and quickly diagnose issues.

In this article, we've explored how to use the APIs provided by ASP.NET to instrument code with signals. We've seen how to use ActivitySource and Activity to create and manage distributed traces, ILogger to log messages with different severity levels, Meter to generate metrics, and Baggage to propagate context across requests.

As you write observable code, keep in mind the following principles:

1. Be intentional: Decide which signals are relevant to your application and when they should be emitted.
2. Be consistent: Use a consistent naming convention for your signals to aid in processing, filtering, and routing.
3. Be selective: Only emit necessary signals to avoid impacting performance and avoid including sensitive or large values in signals.
4. Be proactive: Use signals to monitor your application's behavior and detect issues before they impact users.

By following these principles, you can write observable code that is reliable, scalable, and easy to diagnose.

Feedback

Thank you for reading this article on writing observable code with ASP.NET! I hope you found it informative and useful. As always, I welcome your feedback and insights. Please feel free to leave a comment with your thoughts or questions. Stay tuned for more articles on observability concepts, tools, and techniques!