Introduction to Distributed Tracing

What is Distributed Tracing?

The word tracing is to trace the request as it flows through the system. Since modern systems are distributed where there are micro services interacting with each other in order to fulfil a user request, thus the word distributed, since we are interested in tracing the request in a distributed system formed of interacting services (maybe through API calls or other protocols).

Advantages Distributed Tracing

Between What and How lies another area i.e Why. When we know about what is distributed tracing and before we know how we can achieve it, it equally important to understand why we need it.

Software troubleshooting - Troubleshooting performance issues in a microservice-based architecture is significantly more challenging than in a monolithic software application. Unlike a monolithic application, the root cause of a specific software problem might not be apparent—the overlapping and complex interactions between multiple software modules can make it difficult to diagnose issues.?With distributed tracing, software teams can monitor data that passes through complex paths connecting various micro services and data storages.

Improved Dev Collaboration - Several developers are often involved in building a cloud application, with each responsible for one or several microservices. The software development process slows down if developers cannot trace data exchanged by microservices. With distributed tracing systems developers can collaborate by providing telemetry data, such as logs and traces, for every service request the microservice makes. Thus making it easier and faster to responds to bugs.

Reduce time to market - Organizations deploying distributed tracing platforms can streamline and accelerate efforts to release software applications for end users. As these insights can give more confidence on system performance and easily highlight any bottlenecks for faster correction.

Distributed Tracing Standards

Distributed tracing standards provide a common framework and software tools for developers. By standardising distributed tracing workflow, software teams can instrument request-tracing without being subjected to vendor lock-in.?

Open Telemetry

Before we discuss about open telemetry, its important to understand the building blocks which have been combined to form the standard.

OpenTracing is an open source distributed tracing standard developed by the Cloud Native Computing Foundation (CNCF). OpenTracing focuses on enabling developers to generate traces with an instrumentation API.

OpenCensus consists of multi-language libraries capable of extracting software metrics and sending them to backend systems for analysis. Developers can use the provided API to manage how traces are generated and collected.

OpenTelemetry unifies OpenTracing and OpenCensus. It combines the best features of both standards to provide a comprehensive distributed tracing framework. OpenTelemetry provides extensive software development kits, APIs, libraries, and other instrumentation tools for implementing distributed tracing more effortlessly.?

Key Terms

Next lets take a look at the key terms that would or have appeared in the context of distributed tracing.

Trace - A trace refers to a complete end-to-end path of a request or transaction as it flows through a distributed system. The distributed tracing system assigns a unique ID to every request in order to track it called the trace id.

Span - When processing a service request, an application might take several actions. These actions are represented as spans in distributed tracing. For example, a span might be an API call, user authentication, or enabling storage access. If a single request results in several actions, the initial (or parent) span may branch into several child spans. These nested layers of parent and child spans form a continuous logical representation of steps taken to accomplish the service request. Each span inherits the same trace ID from the original request it belongs to. Spans are also tagged with a unique span ID that helps the tracing system consolidate the metadata, logs, and metrics it collects.?

Context Propagation - Since the traces are distributed, and spans are branching, its important to pass some information between different microservices or components of the system generating traces. Context propagation refers to passing contextual information between different components or services within a distributed system.

Instrumentation Libraries - Instrumentation libraries are software components that developers integrate into their applications to collect tracing data. They are specifically designed to generate, collect, and manage trace data, which includes information about spans and the overall trace. These can automatically capture useful information such as start time, end time, and metadata about each operation (span). They are also responsible for context propagation, ensuring that trace context is passed along with requests as they move through different services in a distributed system. Open telemetry library being one such example helps us achieve the same in vendor agnostic manner.

Data Collectors - These are the components that receive and store trace data, usually in a distributed datastore such as Elasticsearch or Cassandra. For example now Jaeger supports for Jaeger data collector as well as Open Telemetry collector.

Handson!

Lets now build applications that are sending traces to Jaeger. These are spring boot micro services with maven for dependency management.

Before we look into the micro services configurations lets understand a bit about spring boot micrometer and Jaeger.

Spring boot Sleuth and Micrometer

Spring Boot Actuator provides dependency management and auto-configuration for Micrometer Tracing, a facade for popular tracer libraries. In 2016, the Spring Cloud team created a tracing library that could help a lot of developers. It was called Spring Cloud Sleuth. The Spring team realised that tracing could be separated from Spring Cloud and created the Micrometer Tracing project, which is, essentially, a Spring-agnostic copy of Spring Cloud Sleuth.

Micrometer tracer exposes an interface that lets us leverage any bridge or tracing library.

Tracing Library: A library that handles the lifecycle of a span. It can create, start, stop, and report spans to an external system via reporters / exporters.

• Spring Boot Actuator provides dependency management and auto-configuration for Micrometer Tracing.
• When we add a bridge, the micrometer-tracing library is added transitively.
• Currently Micrometer supports two bridges, brave and otel

Here in this example we will leverage otel i.e open telemetry bridge.

Advantage of using spring micrometer is that it provided out of the box handling for HTTP request, DBs Kafka trace and span handling without the need for us to write any code to capture or manage these.

Next level of abstraction is the exporter, we will be leveraging the OpenTelemetry Protocol (is a telemetry data delivery protocol) to export the data to Jaeger, an observability backend that can natively consume the Open telemetry standard data.

Jaeger

Jaeger (originated in Uber) is a Cloud Native Computing Foundation (CNCF) open-source, end-to-end distributed tracing system built for monitoring and troubleshooting microservices-based architectures.

Jaeger vs OpenTelemetry

Jaeger architecture with Open telemetry

Next lets create two spring boot applications, with all required dependencies.

Scenario: We have a service that exposes endpoint to retrieve product information. In order to enrich that info with supplier details, the service depends on another downstream micro service.

ServiceA

This is the client facing microservice. The pom file looks something like this.

It has the dependencies for springboot actuator, otel bridge and otlp exporter (as discussed above).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-parent</artifactId>
       <version>3.4.1</version>
       <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>serviceA</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>serviceA</name>
    <description>Demo project for Spring Boot</description>
    <url/>
    <licenses>
       <license/>
    </licenses>
    <developers>
       <developer/>
    </developers>
    <scm>
       <connection/>
       <developerConnection/>
       <tag/>
       <url/>
    </scm>
    <properties>
       <java.version>17</java.version>
    </properties>
    <dependencies>
       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-web</artifactId>
       </dependency>

       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-actuator</artifactId>
       </dependency>

       <dependency>
          <groupId>io.micrometer</groupId>
          <artifactId>micrometer-tracing-bridge-otel</artifactId>
       </dependency>

       <dependency>
          <groupId>io.opentelemetry</groupId>
          <artifactId>opentelemetry-exporter-otlp</artifactId>
       </dependency>

       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-test</artifactId>
          <scope>test</scope>
       </dependency>
    </dependencies>

    <build>
       <plugins>
          <plugin>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-maven-plugin</artifactId>
          </plugin>
       </plugins>
    </build>

</project>        

The controller for the service is as below

@RestController
@RequestMapping("/v1/products")
public class Controller {

    private static Logger LOGGER = LoggerFactory.getLogger(Controller.class);
    private RestTemplateBuilder restTemplateBuilder;
    private final ProductService productService;

    private RestTemplate restTemplate;

    @Autowired
    public Controller(ProductService productService, RestTemplateBuilder restTemplateBuilder) {
        this.productService = productService;
        this.restTemplate = restTemplateBuilder.build();
    }

    @GetMapping("/{productId}")
    public Product getProduct(@PathVariable String productId) {
        LOGGER.info("going for product id={}", productId);
        Product product = productService.getProduct(productId);
        String uri = "https://localhost:8081/v1/suppliers/"+ product.getSupplier().getId();
        Supplier supplier = restTemplate.getForObject(uri, Supplier.class);
        product.getSupplier().setName(supplier.getName());
        return product;
    }
}        

The service calls the supplier service, simply sets the supplier name in product it fetched from its DB and returns the response.

ProductService

@Component
public class ProductService {

    Map<String, Product> productMap = new HashMap<>();

    public ProductService(){
        productMap.put("123", new Product("123", "ABC", 23, new Supplier("23")));
        productMap.put("124", new Product("124", "ABCC", 06, new Supplier("06")));
    }

    public Product getProduct(String productId) {
        return productMap.get(productId);
    }
}        

Product and Supplier Model

public class Product {
    private String id;
    private String title;
    private Integer price;
    private Supplier supplier;

    public Product(String id, String title, Integer price) {
        this.id = id;
        this.title = title;
        this.price = price;
    }

    public Product(String id, String title, Integer price, Supplier supplier) {
        this.id = id;
        this.title = title;
        this.price = price;
        this.supplier = supplier;
    }

    public String getId() {
        return id;
    }

    public String getTitle() {
        return title;
    }

    public Integer getPrice() {
        return price;
    }

    public Supplier getSupplier() {
        return supplier;
    }

    public void setId(String id) {
        this.id = id;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public void setPrice(Integer price) {
        this.price = price;
    }

    public void setSupplier(Supplier supplier) {
        this.supplier = supplier;
    }
}        

public class Supplier {
    private String id;
    private String name;

    public Supplier(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public String getId() {
        return id;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setId(String id) {
        this.id = id;
    }
}        

application.properties contains the port where traces need to be published. along with sampling rate (i.e how often these should be published).

spring.application.name=serviceA
management.tracing.sampling.probability=1.0
management.otlp.tracing.endpoint=https://localhost:4318/v1/traces        

ServiceB

This is the service that has the supplier related APIs and data.

POM for this is similar to ServiceA

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-parent</artifactId>
       <version>3.4.1</version>
       <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>serviceB</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>serviceB</name>
    <description>Demo project for Spring Boot</description>
    <url/>
    <licenses>
       <license/>
    </licenses>
    <developers>
       <developer/>
    </developers>
    <scm>
       <connection/>
       <developerConnection/>
       <tag/>
       <url/>
    </scm>
    <properties>
       <java.version>17</java.version>
    </properties>
    <dependencies>
       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-web</artifactId>
       </dependency>

       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-actuator</artifactId>
       </dependency>

       <dependency>
          <groupId>io.micrometer</groupId>
          <artifactId>micrometer-tracing-bridge-otel</artifactId>
       </dependency>

       <dependency>
          <groupId>io.opentelemetry</groupId>
          <artifactId>opentelemetry-exporter-otlp</artifactId>
       </dependency>


       <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-test</artifactId>
          <scope>test</scope>
       </dependency>
    </dependencies>

    <build>
       <plugins>
          <plugin>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-maven-plugin</artifactId>
          </plugin>
       </plugins>
    </build>

</project>        

Controller for this service as is below

@RestController
@RequestMapping("/v1/suppliers")
public class Controller {

    private final SupplierService supplierService;

    @Autowired
    public Controller(SupplierService supplierService) {
        this.supplierService = supplierService;
    }

    @GetMapping("/{supplierId}")
    public Supplier getSupplier(@PathVariable String supplierId) {
        return supplierService.getSupplier(supplierId);
    }
}        

SupplierService

@Component
public class SupplierService {

    Map<String, Supplier> supplierMap = new HashMap<>();

    public SupplierService(){
        supplierMap.put("23", new Supplier("Aneshka"));
        supplierMap.put("06", new Supplier("ABC"));
    }

    public Supplier getSupplier(String Id){
        return supplierMap.get(Id);
    }
}        

Supplier Model for ServiceB

public class Supplier {
    private String name;

    public Supplier(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}        

application.properties file for this service is similar to service A.

spring.application.name=serviceB
server.port=8081
management.tracing.sampling.probability=1.0
management.otlp.tracing.endpoint=https://localhost:4318/v1/traces        

Jaeger Setup

We would be running Jaeger in a container, managed by podman. We could even leverage docker to run the container and commands are similar mostly podman term is replaced with docker.

podman run -p 16686:16686 -p 4317:4317 -p 4318:4318 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:latest        

The environment variable lets the Jaeger collect OTLP traces directly. The port 16686 will be used to access Jaeger UI locally. 4318 is the port to which our application publishes(leveraging REST over HTTP) the data and 4317 is for gRPC protocol.

Once we run both the services this is how the trace looks like for GET endpoint

https://localhost:8080/v1/products/123        

Summary

In this blog we discussed about what is distributed tracing, why should be looking towards adopting distributed tracing, it easy adoption with minimal dependencies when we are working with Spring boot framework. We also looked at Open telemetry as distributed tracing standard which is leveraged to achieve tracing without vendor lockin. At last with a simple hands on we were able to develop and see how tracing works with a well known backend Jaeger, out of the box for basic Rest bases calls.

Sources of Knowledge

https://aws.amazon.com/what-is/distributed-tracing/

https://opentelemetry.io/docs/what-is-opentelemetry/

https://spring.io/blog/2022/10/12/observability-with-spring-boot-3

https://www.jaegertracing.io/docs/1.23/architecture/