Practical observability: distributed tracing with otel4s

In this series, we explore how to bring observability into a Scala application using OpenTelemetry - the open standard for telemetry data - and otel4s, a purely functional OpenTelemetry library for the Typelevel (Cats Effect) ecosystem.

This article explains how to implement clear and predictable distributed tracing in Scala with otel4s.

The early sections outline how telemetry signals are identified by resources, instrumentation scopes, and attributes. They also explain how tracers and tracer providers define boundaries and lifecycle, providing context for span behavior and the otel4s API structure.

Then we focus on span creation, scoping, and context propagation, detailing how traces cross service boundaries. These concepts are later demonstrated in a real application to show how explicit instrumentation affects observed traces.

1. Identifying telemetry signals

When you look at a trace in Grafana Tempo or Jaeger, you're not just viewing timings. You're seeing information from different parts of an application, compiled into a single view. Understanding where that information comes from makes the tracing API much easier to reason about.

OpenTelemetry defines a consistent identification model for all telemetry. Each signal has a process-level identity, a scope-level identity, and operation-level details. These aspects are kept separate and clearly defined.

The tracing backends, like Grafana Tempo or Jaeger, rely on this information. It determines which service emitted the signal, which library or subsystem produced it, and what kind of operation it represents.

1.1. Attributes

Attributes are key-value pairs used to describe telemetry data. They are a core concept in OpenTelemetry and are used with traces, metrics, and logs.

An attribute provides structured context to a signal. In traces, attributes describe what a span represents. In metrics, they label time series. In logs, they offer structured fields. The same attribute key, such as a service.name or http.request.method, can appear consistently across all three signals.

Because attributes are structured, backends can index them. This enables filtering traces by route, grouping metrics by operation, or correlating logs with a specific request. Without attributes, telemetry becomes difficult to query and analyze.

In tracing, span attributes typically capture operation-level details, such as the HTTP method, route template, database system, or request type. These values explain what kind of work the span represents, rather than how it was executed.

1.2. Instrumentation resource

A resource is a process-level identity that defines the entity generating telemetry. In practice, it describes the running service with attributes like service.name, service.version, or deployment.environment.

Resource attributes are attached to all telemetry produced by a specific SDK instance. They are intended to remain constant for the lifetime of the process and do not vary per request.

In otel4s, you configure the resource when creating the SDK, generally through environment variables or system properties.

1.3. Instrumentation scope

The instrumentation scope is a scope-level identity that describes the library or module that emitted the telemetry signal. Concretely, it is defined by a name, an optional version, and an optional schema URL.

In otel4s, this information is attached when you create a Tracer. The tracer name usually corresponds to a library, a module, or a logical subsystem. For application code, it is common to use something like module.forecast or module.payments.

Backends use the instrumentation scope to group spans, filter noise, and reason about ownership. When you later see traces with spans from an HTTP server, a database client, and a Kafka consumer, the instrumentation scope helps you tell them apart.

---
config:
  look: handDrawn
  layout: elk
---
graph TB
  subgraph R["Telemetry resource<br/>(process wide)"]
    Rdesc["Attributes:<br/>service.name<br/>service.version<br/>deployment.environment"]

    subgraph S["Instrumentation scope (library or module)"]
      Sdesc["name<br/>version<br/>schema URL"]

      subgraph SIG["Telemetry signal"]
        Span["Trace / Metric / Log record"]
        Attr["Attributes:<br/>operation.type<br/>http.request.method<br/>user.id"]
        Span --> Attr
      end
    end

end

Telemetry signal identification layers

The telemetry resource identifies the running service or process for all SDK output. The instrumentation scope refers to a specific library, module, or subsystem.

A span, metric point, or log record always belongs to a single instrumentation scope and the telemetry resource. Attributes are attached at the signal level. They describe what the specific span, metric, or log represents and serve as the main mechanism for querying, grouping, and correlating telemetry.

This nesting, from resource to instrumentation scope to individual signals, explains why OpenTelemetry can maintain ownership, identity, and meaning separation while still presenting everything as a single, coherent trace or view in the backend.

2. Initializing the SDK

Before creating tracers or spans, an SDK must be initialized. The instrumentation API used by the application is exposed through this instance.

In most applications, you create one instance at startup. You keep it for the entire lifetime of the process. Treat it as a shared dependency to make ownership clear and keep configuration centralized.

First, we need to add a few dependencies and configuration options:

libraryDependencies ++= Seq(
  "org.typelevel"   %% "otel4s-oteljava"                           % "0.15.0",
  "io.opentelemetry" % "opentelemetry-exporter-otlp"               % "1.58.0" % Runtime,
  "io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % "1.58.0" % Runtime,
)
run / fork  := true
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true"
javaOptions += "-Dotel.service.name=weather-service"

//> using dep "org.typelevel::otel4s-oteljava:0.15.0"
//> using dep "io.opentelemetry:opentelemetry-exporter-otlp:1.58.0"
//> using dep "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.58.0"
//> using javaOpt "-Dotel.java.global-autoconfigure.enabled=true"
//> using javaOpt "-Dotel.service.name=weather-service"

Here we add otel4s with the oteljava backend, which uses the official OpenTelemetry Java SDK under the hood. We include the SDK autoconfiguration module and the OTLP exporter. These components allow the SDK to configure resources, exporters, and sampling using environment variables and JVM (system) properties rather than hard-coded values.

With the dependencies in place, we can create an autoconfigured instance:

import cats.effect.{IO, IOApp}
import org.typelevel.otel4s.Otel4s
import org.typelevel.otel4s.oteljava.OtelJava
import org.typelevel.otel4s.trace.TracerProvider

object Main extends IOApp.Simple {
  def run: IO[Unit] =
    OtelJava.autoConfigured[IO]().use { otel4s =>
      given TracerProvider[IO] = otel4s.tracerProvider

      program
    }

  def program(using TracerProvider[IO]): IO[Unit] =
    IO.println("My program")
}

OtelJava.autoConfigured builds an SDK instance using environment variables, JVM properties, and defaults. It returns a Resource, so the lifecycle is managed for you. When released, exporters flush and shut down gracefully.

2.1. Autoconfigured versus global

otel4s offers two main ways to integrate with the OpenTelemetry Java SDK.

The first option is OtelJava.autoConfigured, the default for most applications. It creates an isolated, non-global SDK instance configured by the environment. Exporters, sampling strategies, resource detectors, and context storage are automatically configured.

This instance is not registered as the global OpenTelemetry SDK. Multiple autoconfigured instances are completely independent and do not see each other's spans. Usually, you create a single instance for the lifetime of the process. This model suits functional applications built with IOApp, where resource ownership and shutdown are explicit.

OtelJava.autoConfigured also accepts an optional customization function. This allows you to tweak the underlying SDK builder while still relying on autoconfiguration for most settings:

OtelJava.autoConfigured[IO] { builder =>
  builder.addResourceCustomizer { (resource, _) =>
    resource.toBuilder.put("custom.attribute", "value").build()
  }
}

The second option is OtelJava.global, which configures otel4s to use the global OpenTelemetry instance. The global SDK must be initialized before OtelJava.global is materialized. Because otel4s does not manage the lifecycle, the owner of the global SDK is responsible for flushing and shutting down.

This approach is useful when an IO-based code runs inside a larger application that already relies on a global OpenTelemetry setup, and you want otel4s to participate in it.

2.2. Configuration through the environment

Autoconfiguration uses environment variables and JVM options. For example, variables like OTEL_TRACES_EXPORTER select exporters, OTEL_EXPORTER_OTLP_ENDPOINT sets endpoints, and OTEL_TRACES_SAMPLER chooses sampling strategies.

By default, the SDK builds a telemetry resource using detectors. These usually include process information, host details, and any attributes provided through OTEL_RESOURCE_ATTRIBUTES.

In a minimal setup, the only required configuration is the service name. You can provide it using the otel.service.name system property or the OTEL_SERVICE_NAME environment variable. All other settings can be configured later without modifying application code.

This approach scales across environments. You can deploy the same service to development, staging, or production with varied observability. Traces remain structurally identical. Differences in origin, environment, or region are reflected in telemetry resource attributes, not in code paths.

Here are a few examples of common settings:

You can check the full list of supported settings in the OpenTelemetry Java SDK documentation.

3. Creating a Tracer

---
config:
  look: handDrawn
  layout: elk
---
graph TB
subgraph R["OtelJava<br/>(OpenTelemetry Java SDK)"]

    subgraph TracerProvider
      subgraph Tracer
        TracerApi@{ shape: processes, label: "Spans" }
        TracerScope["Instrumentation scope:<br/>name<br/>version<br/>schema URL"]
      end
    end

    Pipeline["Telemetry Pipeline"]

    TracerProvider --> Pipeline
end

Components hierarchy

The TracerProvider is the main entry point for tracing in otel4s. It produces Tracer instances for different instrumentation scopes and connects them to the telemetry pipeline. We can get it from the OtelJava instance.

A Tracer represents a single instrumentation scope. Every span is created through a tracer, and the tracer defines which part of the system those spans belong to. In otel4s, tracers are cheap to create and safe to reuse. You typically create one tracer per module or logical group and make it available wherever spans are needed.

Let's start with a simple example:

import cats.effect.IO
import org.typelevel.otel4s.trace.{Tracer, TracerProvider}

class ForecastModule(using Tracer[IO]) {
  def checkWeather(location: String): IO[String] =
    getForecast(location)

  private def getForecast(location: String): IO[String] =
    IO.println(s"Getting forecast for $location").as("Sunny")
}

object ForecastModule {
  def create(using TracerProvider[IO]): IO[ForecastModule] =
    for {
      given Tracer[IO] <- TracerProvider[IO].tracer("module.forecast").withVersion("1.2.3").get
    } yield new ForecastModule
}

Here, we create a Tracer named module.forecast with version 1.2.3. These options identify the instrumentation scope. In application code, the name typically corresponds to a logical subsystem, such as a domain module, an HTTP layer, or a background worker.

The version becomes useful when traces from multiple deployments overlap. If span names or attributes change between releases, the version tells you which code produced which spans. This makes it easier to reason about traces during rollouts. It also helps when comparing behavior across versions.

The tracer is created once and passed to the module implicitly. All spans produced by ForecastModule now carry the same scope information. This makes their origin explicit in the trace.

A common question at this stage is whether an application should use a single tracer or multiple tracers.

There's no strict rule, but a useful guideline: use separate tracers for spans from different layers and use a single tracer if spans belong to one cohesive unit.

For example, use one tracer for an HTTP layer and another for a domain logic. This helps visually separate transport concerns from business behavior in a trace. Splitting tracers too aggressively, however, can add noise without improving clarity.

4. Creating a Span

A span is a unit of work with a name, timing, optional attributes and events, and trace context. In otel4s, spans are always created via a Tracer to ensure proper instrumentation scope and context association.

Tracer[IO].span is the main API for application code, handling span creation, context management, and cleanup in a single step, compatible with effectful code.

A minimal example looks like this:

import cats.effect.IO
import org.typelevel.otel4s.trace.{StatusCode, Tracer}
import org.typelevel.otel4s.Attribute

class ForecastModule(using Tracer[IO]) {
  def checkWeather(location: String): IO[String] =
    Tracer[IO].span("checkWeather", Attribute("location", location)).use { span =>
      getForecast(location) <* span.addEvent("forecast-received")
    }

  private def getForecast(location: String): IO[String] =
    IO.println(s"Getting forecast for $location").as("Sunny")
}

This code creates a span named checkWeather, makes it current for the duration of the effect, and guarantees that it is ended when the effect completes.

Several important things happen here.

First, parent selection is automatic: if there's an active span, the new one becomes a child; if not, a new trace starts. This establishes the parent-child relationship that forms the trace structure.

Second, error awareness is built in. If the effect fails, the span is marked as errored, and the exception is recorded. You do not need to catch exceptions just to annotate the span or remember to set status codes on every failure path. Span outcome follows effect outcome by default.

When the effect finishes, the previous context is restored. This maintains span nesting and prevents context leaks.

4.1. Nesting spans

When a span is created inside another span, the inner span automatically becomes a child of the outer one. You do not need to manage scopes explicitly or pass span references through method signatures.

For example:

class ForecastModule(using Tracer[IO]) {
  def checkWeather(location: String): IO[String] =
    Tracer[IO].span("checkWeather", Attribute("location", location)).use { span =>
      getForecast(location) <* span.setStatus(StatusCode.Ok)
    }

  private def getForecast(location: String): IO[String] =
    Tracer[IO].span("getForecast").surround {
      IO.println(s"Getting forecast for $location").as("Sunny")
    }
}

The resulting trace links getForecast as a child of checkWeather, preserving timing and structure. Context propagates automatically across calls, fibers, and async bounds within the Cats Effect runtime.

Span nesting reflects effect structure, not thread structure. This is critical in fiber-based systems, where execution can switch threads often.

4.2. Accessing the current span

While a span is active, it is available through the current tracing context. This allows nested code to interact with an existing span without owning its lifecycle:

private def getForecast(location: String): IO[String] =
  IO.println(s"Getting forecast for $location").as("Sunny") <*
  Tracer[IO].withCurrentSpanOrNoop(_.setStatus(StatusCode.Ok)) 

This code doesn't create a span. It retrieves the current one from the scope and updates it if present. If no span is active, a no-op span is used, and setStatus has no effect.

This pattern keeps span ownership at higher levels while still allowing deeper layers to report outcomes, add attributes, or emit events. Domain code remains loosely coupled to tracing and does not need to know how spans are created or finalized.

4.3. The SpanBuilder deep dive

Tracer[IO].span covers typical cases, managing lifecycle, context propagation, and errors with minimal API surface.

At system boundaries, you often need advanced control over span creation. Typical scenarios include integrating with HTTP servers, gRPC handlers, or message consumers, where span creation must align with context or protocols provided by external systems.

For these cases, otel4s exposes a low-level builder API via Tracer[IO].spanBuilder:

import org.typelevel.otel4s.trace.*

def withSpanBuilder[A](externalParent: SpanContext, linkTo: SpanContext)(io: IO[A])(using Tracer[IO]): IO[A] = {
  val spanOps: SpanOps[IO] = Tracer[IO]
    .spanBuilder("span.name")
    .addAttributes(Attribute("request.type", "example"))
    .addLink(linkTo)
    .withParent(externalParent)
    .withSpanKind(SpanKind.Internal)
    .withFinalizationStrategy(SpanFinalizer.Strategy.reportAbnormal)
    .build

  spanOps.surround(io)
}

The builder makes span creation explicit and configurable. It supports overriding parents, attaching links, setting the span kind, and customizing finalization. These capabilities are intentionally excluded from the high-level API because the business layer rarely needs them, whereas boundary code often does.

Here is the full list of methods available on the builder:

Use the builder API for advanced integration points where you need to align spans with externally provided context, correlate work between systems, or customize span attributes or timing. For typical application code, continue using the high-level API for ease of use.

4.4. The context propagation model

You may have noticed that Tracer[IO].spanBuilder(...).build method returns a SpanOps[IO], not a Resource[IO, Span[IO]].

Span creation and span propagation management are closely related. A span is not just a value that needs to be acquired and released. It also needs to be made current so downstream operations can attach to it. This requires careful control over scope.

Resource is a good fit for managing lifecycles, but it does not model dynamic context propagation well. Resource acquisition and release can happen on different fibers, which breaks the assumption that a span should be current only for a specific region of execution.

otel4s solves this by basing the context propagation on cats.mtl.Local:

trait Local[F[_], E] {
  def ask: F[E]
  def local[A](fa: F[A])(f: E => E): F[A]
}

Instead of binding to threads, context is bound to effect segments. Local ensures the same context is observed throughout execution of a segment, regardless of how the fiber is scheduled. This is what makes span nesting and access to the current span work transparently across asynchronous boundaries, without passing Span[IO] explicitly or relying on thread locals.

SpanOps bridges this gap by combining lifecycle management with scoped context propagation. It wraps effects, ensuring tracing context is installed and restored correctly.

This design ensures the tracing API is predictable and safe for concurrent and async code.

5. Explicit scope manipulation

By default, Tracer[IO].span determines parentage automatically based on the currently active tracing context.

Sometimes the default behavior is not enough. You may want to suppress tracing for a block of code, start a new trace, or attach work to a specific parent that is not currently active. These cases often occur at boundaries where execution semantics change and implicit propagation no longer matches intent.

otel4s provides explicit scope controls for these situations. The intent of a scope is to temporarily redefine the current tracing context, directly affecting how spans created within it are parented. Rather than creating spans, scopes determine the parenting relationships for spans created during their lifetime.

The following example demonstrates how these scopes affect span parentage in practice:

def scopeManagement()(using Tracer[IO]): IO[Unit] =
  Tracer[IO].span("parent").use { parent =>
    for {
      _ <- Tracer[IO].span("child-1").use_ // a child of the 'parent'
      _ <- Tracer[IO].rootScope {
             Tracer[IO].span("child-2").use_ // a root span that is not associated with the 'parent'
           }
      _ <- Tracer[IO].noopScope {
             Tracer[IO].span("child-3").use_ // a span will not be created
           }
      _ <- Tracer[IO].span("child-4").surround { // a child of the 'parent'
             Tracer[IO].childScope(parent.context).surround {
               Tracer[IO].span("explicit-child").use_ // a child of the 'parent', not 'child-4'
             }
           }
    } yield ()
  }

The first span, child-1, is a straightforward child of parent, created under the default propagation rules.

Within rootScope, child-2 becomes a root span. This breaks the default continuity, as parent is still active outside but not within the new scope. The explicit scope here signals the intent to start a new trace context.

Inside noopScope, no span is created. Tracing has no effect here, letting you suppress traces without extra logic.

The last case shows scope interaction. child-4 follows parent by normal propagation. Inside, childScope(parent) overrides the scope, so explicit-child is a direct child of parent, not child-4. This demonstrates that scope defines intent, apart from the execution structure.

This level of control is powerful but easy to misuse. Overusing explicit scopes can make traces harder to interpret, as parent-child relationships no longer reflect the execution flow. Scope manipulation should mark meaningful execution boundaries. Use it where execution meaningfully changes, not as a replacement for the default propagation model.

6. Cross-service propagation

Up to this point, everything has happened within a single process. Spans are created, nested, and propagated automatically as execution moves across fibers. This model holds as long as all work stays within one runtime.

The moment execution crosses a process boundary, the rules change. A span cannot follow execution into another process. Instead, only the tracing context is transferred. This happens at explicit boundaries where a service hands work off to something else, such as an HTTP call, a gRPC request, or a message being published or consumed.

Tracing context represents a small data structure containing the trace ID, parent span ID, and sampling flags. On the caller side, the tracing context is encoded into request metadata. On the receiver side, it is decoded and set as the current context, allowing new spans to join the same trace. If decoding succeeds, the trace continues. If decoding fails, execution proceeds under a new root context.

Encoding and decoding operations are performed by propagators. A propagator specifies the representation of tracing context as a set of key-value pairs and defines its reconstruction on the receiving side.

By default, the OpenTelemetry SDK uses the W3C TraceContext propagator. Propagators are configured at the SDK level, typically using environment variables or JVM system properties, rather than in application code.

With the W3C TraceContext propagator, injected headers look like this:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

where 4bf92f3577b34da6a3ce929d0e0e4736 is the trace ID, 00f067aa0ba902b7 is the span ID, and 01 is the trace flags.

6.1. Outgoing side: propagating context

On the outgoing side, the goal is to attach the current tracing context to an outbound request.

In otel4s, this starts with Tracer[IO].propagate, which reads the current tracing context and uses the configured propagators to encode it as text key-value pairs. These pairs are written into a carrier.

A carrier is a data structure that holds string keys and values. At this point, nothing is sent over the wire. No transport is involved yet. The carrier is simply a container for the encoded tracing context.

The second step is transport-specific. Once you have a populated carrier, you attach it to an outgoing request using whatever mechanism the protocol provides, such as HTTP headers, gRPC metadata, or message headers.

Tracer[IO].propagate works with any carrier that implements TextMapUpdater. This typeclass defines how key-value pairs are written into a carrier. As long as a data structure can store textual keys and values, it can be used for propagation.

For example, this is how propagation can be wired into org.http4s.Headers:

import cats.effect.IO
import org.http4s.{Headers, Request}
import org.http4s.client.Client
import org.typelevel.otel4s.trace.Tracer
import org.typelevel.otel4s.context.propagation.TextMapUpdater

def handleOutgoing(request: Request[IO], client: Client[IO])(using Tracer[IO]): IO[String] =
  for {
    headers  <- Tracer[IO].propagate(Headers.empty)
    response <- client.expect[String](request.withHeaders(headers))
  } yield response

given TextMapUpdater[Headers] with {
  def updated(carrier: Headers, key: String, value: String): Headers =
    carrier.put(key -> value)
}

Here, the current tracing context is written into the Headers instance, which is then attached to the outgoing request. Downstream services can extract this context and continue the trace.

6.2. Incoming side: extracting context

On the incoming side, the goal is the inverse. Tracing context must be read from request metadata, decoded using the configured propagators, and installed as the current context while the request is handled.

In otel4s, this is done with Tracer[IO].joinOrRoot. This call reads encoded context from a carrier, attempts to decode it, and establishes the resulting context as current for the duration of the effect.

As a result, if decoding succeeds, spans created inside the block become children of the upstream span. If decoding fails, execution continues under a fresh root context, and the first span you create becomes the root of a new trace.

As on the outgoing side, the same carrier is extracted from the request, and, as before, it is not tied to any specific transport.

Tracer[IO].joinOrRoot also works with any carrier type that implements TextMapGetter, a typeclass that defines how key-value pairs are read from a carrier.

For example, this is how context can be extracted directly from org.http4s.Headers:

import cats.effect.IO
import org.http4s.{Headers, Request, Response}
import org.typelevel.ci.CIString
import org.typelevel.otel4s.context.propagation.TextMapGetter
import org.typelevel.otel4s.trace.{SpanKind, Tracer}

def handleIncoming(req: Request[IO])(
  handler: Request[IO] => IO[Response[IO]]
)(using Tracer[IO]): IO[Response[IO]] =
  Tracer[IO].joinOrRoot(req.headers) {
    Tracer[IO]
      .spanBuilder(s"${req.method} ${req.uri}")
      .withSpanKind(SpanKind.Server)
      .build
      .surround(handler(req))
  }

given TextMapGetter[Headers] with {
  def get(carrier: Headers, key: String): Option[String] =
    carrier.get(CIString(key)).map(_.head.value)

  def keys(carrier: Headers): Iterable[String] =
    carrier.headers.view.map(_.name).distinct.map(_.toString).toSeq
}

Here, context extraction happens exactly once at the boundary. After that, normal in-process propagation takes over. The server span is explicitly created using the extracted or fresh context, and all downstream spans automatically attach to it.

This boundary-focused approach keeps propagation explicit and predictable. Context is injected only when leaving the process and extracted only when entering it. Everything else relies on the same propagation rules you already use inside a single service.

7. Instrumenting an application

Now we apply the earlier concepts to a concrete service: resources and scopes identify layers, and propagation occurs only at process boundaries, while the rest of the code relies on in-process context flow.

This section applies tracing to a small weather service that fetches the forecast for a given location. The goal is to show how tracing fits into real code through clear boundaries, deliberate span placement, and explicit wiring.

The example service has an HTTP endpoint that returns a weather forecast, split into a business module and an HTTP layer. It is intentionally small, but the structure scales to larger services. Tracing reflects this design: domain spans are placed around service actions within the business module, while HTTP spans are positioned around request handling and context propagation in the HTTP layer.

For this example, the dependencies and JVM options look like this:

scalaVersion := "3.7.4"
libraryDependencies ++= Seq(
  "org.http4s"       %% "http4s-ember-client"                      % "0.23.33",
  "org.http4s"       %% "http4s-ember-server"                      % "0.23.33",
  "org.http4s"       %% "http4s-dsl"                               % "0.23.33",
  "org.typelevel"    %% "otel4s-oteljava"                          % "0.15.0",
  "org.typelevel"    %% "otel4s-semconv"                           % "0.15.0",
  "io.opentelemetry" % "opentelemetry-exporter-otlp"               % "1.58.0",
  "io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % "1.58.0",
  "ch.qos.logback"   % "logback-classic"                           % "1.5.24",
)
run / fork  := true
javaOptions += "-Dotel.service.name=weather-service"
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true"

//> using scala "3.7.4"
//> using dep "org.http4s::http4s-ember-client:0.23.33"
//> using dep "org.http4s::http4s-ember-server:0.23.33"
//> using dep "org.http4s::http4s-dsl:0.23.33"
//> using dep "org.typelevel::otel4s-oteljava:0.15.0"
//> using dep "org.typelevel::otel4s-semconv:0.15.0"
//> using dep "io.opentelemetry:opentelemetry-exporter-otlp:1.58.0"
//> using dep "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.58.0"
//> using dep "ch.qos.logback:logback-classic:1.5.24"
//> using javaOpt "-Dotel.service.name=weather-service"
//> using javaOpt "-Dotel.java.global-autoconfigure.enabled=true"

7.1. Instrumenting the business layer

In the business layer, tracing describes domain behavior by placing spans around key operations. Use attributes and events for domain context. This layer ignores headers, carriers, and propagation, assuming tracing context exists and focusing only on clearly demarcated units of work.

Business logic sits in a dedicated ForecastModule with its own tracer and scope. The tracer, created once, identifies the module as domain code. Business spans stand apart from HTTP spans in traces and views, making ownership clear.

class ForecastModule(client: Client[IO])(using Tracer[IO]) {
  def checkWeather(location: String): IO[String]
}

object ForecastModule {
  def create(client: Client[IO])(using TracerProvider[IO]): IO[ForecastModule] =
    for {
      given Tracer[IO] <- TracerProvider[IO].tracer("module.forecast").withVersion("1.2.3").get
    } yield new ForecastModule(client)
}

The business layer spans define the domain-level units of work. In this module, checkWeather is the public entry point, so a span is placed around it as the top-level business span.

def checkWeather(location: String): IO[String] =
  Tracer[IO].span("checkWeather", Attribute("location", location)).use { span =>
    getForecast(location) <* span.addEvent("forecast-received")
  }

Span boundaries are chosen based on domain meaning rather than implementation details. Even if checkWeather later grows to include caching, retries, or additional calls, the span boundary remains correct.

Attributes capture stable domain inputs. The location attribute is attached to checkWeather because it is central and helps with grouping and filtering.

Events mark notable moments within a span without adding another span. The forecast-received event records a meaningful completed step.

The getForecast method is traced separately because it represents a distinct unit of work within the operation.

private def getForecast(location: String): IO[String] =
  Tracer[IO].span("getForecast").use { span =>
    for {
      forecast <- client.expect[String](s"https://wttr.in/$location?format=3")
      _        <- span.setStatus(StatusCode.Ok)
    } yield forecast
  }

Span status reflects the outcome of a domain operation. Explicitly setting status helps when success or failure is part of a normal control flow and should be immediately visible in the trace.

The business layer does not manage propagation, cancellation, or lifecycle aspects. It assumes context is already established and focuses only on describing domain work.

import cats.effect.IO
import org.http4s.client.Client
import org.typelevel.otel4s.Attribute
import org.typelevel.otel4s.trace.{StatusCode, Tracer, TracerProvider}

class ForecastModule(client: Client[IO])(using Tracer[IO]) {
  def checkWeather(location: String): IO[String] =
    Tracer[IO].span("checkWeather", Attribute("location", location)).use { span =>
      getForecast(location) <* span.addEvent("forecast-received")
    }

  private def getForecast(location: String): IO[String] =
    Tracer[IO].span("getForecast").use { span =>
      for {
        _        <- IO.println(s"Getting forecast for $location")
        forecast <- client.expect[String](s"https://wttr.in/$location?format=3")
        _        <- span.setStatus(StatusCode.Ok)
      } yield forecast
    }
}

object ForecastModule {
  def create(client: Client[IO])(using TracerProvider[IO]): IO[ForecastModule] =
    for {
      given Tracer[IO] <- TracerProvider[IO].tracer("module.forecast").withVersion("1.2.3").get
    } yield new ForecastModule(client)
}

7.2. Instrumenting the HTTP layer

At the HTTP layer, tracing can be centralized at the middleware level. This ensures consistent request lifecycle tracing without duplicating manual span creation across routes.

On the server, it extracts context, creates server spans, and adds protocol attributes. On the client, it creates client spans and propagates context for outbound calls.

This approach ensures every request is traced consistently and protocol-specific aspects remain at the HTTP layer.

7.2.1. HTTP server

The server is implemented as a dedicated HttpServer with its own tracer and instrumentation scope. Spans created here represent transport-level work and the lifecycle of a request, not domain logic.

class HttpServer(forecastModule: ForecastModule)(using Tracer[IO]) {
  def start: Resource[IO, Server]
  def httpApp: HttpApp[IO]
}

object HttpServer {
  def create(forecastModule: ForecastModule)(using TracerProvider[IO]): IO[HttpServer] =
    for {
      given Tracer[IO] <- TracerProvider[IO].get("org.http4s.server")
    } yield new HttpServer(forecastModule)
}

Tracing middleware is applied to the entire HttpApp, wrapped in IO.uncancelable to ensure the server span is always finalized, even if a request is canceled. This prevents scenarios where canceled requests end tracing prematurely, leaving spans incomplete and creating diagnostic blind spots. Actual request handling runs inside the poll block and remains cancelable.

class HttpServer(forecastModule: ForecastModule)(using Tracer[IO]) {
  private def tracingMiddleware(httpApp: HttpApp[IO]): HttpApp[IO] =
    Kleisli { (req: Request[IO]) =>
      IO.uncancelable { poll =>
        Tracer[IO].joinOrRoot(req.headers) {
          Tracer[IO]
            .spanBuilder(s"${req.method} ${req.uri}")
            .withSpanKind(SpanKind.Server)
            .build
            .use { span =>
              poll(httpApp.run(req)).flatTap { response =>
                span.addAttribute(HttpAttributes.HttpResponseStatusCode(response.status.code))
              }
            }
        }
      }
    }
}

Tracer[IO].joinOrRoot establishes the parent context at the boundary. Everything downstream relies on normal in-process propagation. The server span covers the full request lifecycle, and protocol attributes are added once the response is available.

Routes themselves focus only on request handling and domain logic. They do not explicitly create spans or manage context.

private def routes: HttpRoutes[IO] = HttpRoutes.of {
  case req @ GET -> Root / "weather" / location =>
    for {
      forecast <- forecastModule.checkWeather(location)
    } yield Response[IO]().withEntity(forecast)
}

import cats.data.Kleisli
import cats.effect.{IO, Resource}
import org.http4s.dsl.io.*
import org.http4s.*
import org.http4s.server.Server
import org.http4s.ember.server.EmberServerBuilder
import org.typelevel.ci.CIString
import org.typelevel.otel4s.Attribute
import org.typelevel.otel4s.context.propagation.TextMapGetter
import org.typelevel.otel4s.semconv.attributes.HttpAttributes
import org.typelevel.otel4s.trace.*

class HttpServer(forecastModule: ForecastModule)(using Tracer[IO]) {
  import HttpServer.given

  def start: Resource[IO, Server] =
    EmberServerBuilder.default[IO].withHttpApp(httpApp).build

  def httpApp: HttpApp[IO] =
    tracingMiddleware(routes.orNotFound)

  private def routes: HttpRoutes[IO] = HttpRoutes.of { 
    case req @ GET -> Root / "weather" / location =>
      for {
        forecast <- forecastModule.checkWeather(location)
      } yield Response[IO]().withEntity(forecast)
  }

  private def tracingMiddleware(httpApp: HttpApp[IO]): HttpApp[IO] =
    Kleisli { (req: Request[IO]) =>
      IO.uncancelable { poll =>
        Tracer[IO].joinOrRoot(req.headers) {
          Tracer[IO]
            .spanBuilder(s"${req.method} ${req.uri}")
            .withSpanKind(SpanKind.Server)
            .build
            .use { span =>
              poll(httpApp.run(req)).flatTap { response =>
                span.addAttribute(HttpAttributes.HttpResponseStatusCode(response.status.code))
              }
            }
        }
      }
    }

}

object HttpServer {
  def create(forecastModule: ForecastModule)(using TracerProvider[IO]): IO[HttpServer] =
    for {
      given Tracer[IO] <- TracerProvider[IO].get("org.http4s.server")
    } yield new HttpServer(forecastModule)

  private given TextMapGetter[Headers] with {
    def get(carrier: Headers, key: String): Option[String] =
      carrier.get(CIString(key)).map(_.head.value)

    def keys(carrier: Headers): Iterable[String] =
      carrier.headers.view.map(_.name).distinct.map(_.toString).toSeq
  }
}

7.2.2. HTTP client

The HTTP client is also instrumented as a separate module with its own tracer and instrumentation scope. Client spans represent outbound calls, and the client is responsible for injecting the active tracing context into request headers.

object HttpClient {
  def create(using TracerProvider[IO]): Resource[IO, Client[IO]]
}

The middleware creates a span for each request, propagates context, and records response attributes. It is wrapped in Resource.uncancelable to ensure the span is finalized even if the caller cancels the request.

private def tracingMiddleware(client: Client[IO])(using Tracer[IO]): Client[IO] =
  Client { request =>
    Resource.uncancelable { poll =>
      for {
        spanOps <- Tracer[IO]
          .spanBuilder(s"${request.method} ${request.uri}")
          .withSpanKind(SpanKind.Client)
          .build
          .resource

        headers <- Resource.eval(Tracer[IO].propagate(Headers.empty))
        response <- poll(client.run(request.withHeaders(headers)).mapK(spanOps.trace))

        _ <- Resource.eval(
          spanOps.span.addAttribute(HttpAttributes.HttpResponseStatusCode(response.status.code))
        )
      } yield response
    }
  }

Tracer[IO].propagate encodes the active span context into request headers so that downstream services can join the same trace. Client spans are named by HTTP method and URI, and the response status code is recorded upon completion of the call.

The value returned by .resource is a SpanOps.Res, which contains both the span itself and a trace function. This distinction is important because span propagation does not happen automatically through Resource.

This matters because the client.run returns a Resource[IO, Response[IO]]. By applying mapK(spanOps.trace), we transform the underlying effect so that all actions performed while using the response, including reading the body, run inside the span context. Without this step, the span would start and end correctly, but the actual HTTP work would not appear under the client span in the trace.

This pattern ensures that the client span accurately represents the full lifecycle of the outbound request, from sending headers to consuming the response, while keeping propagation and lifecycle management localized to the HTTP layer.

import cats.effect.{IO, Resource}
import org.http4s.*
import org.http4s.client.Client
import org.http4s.client.middleware.Logger
import org.http4s.ember.client.EmberClientBuilder
import org.typelevel.otel4s.context.propagation.TextMapUpdater
import org.typelevel.otel4s.semconv.attributes.HttpAttributes
import org.typelevel.otel4s.trace.*

object HttpClient {

  def create(using TracerProvider[IO]): Resource[IO, Client[IO]] =
    EmberClientBuilder
      .default[IO]
      .build
      .map(client => Logger[IO](logHeaders = true, logBody = true)(client))
      .evalMap(client => HttpClient.traced(client))

  private def traced(client: Client[IO])(using TracerProvider[IO]): IO[Client[IO]] =
    for {
      given Tracer[IO] <- TracerProvider[IO].get("org.http4s.client")
    } yield tracingMiddleware(client)

  private def tracingMiddleware(client: Client[IO])(using Tracer[IO]): Client[IO] =
    Client { request =>
      Resource.uncancelable { poll =>
        for {
          spanOps <- Tracer[IO]
            .spanBuilder(s"${request.method} ${request.uri}")
            .withSpanKind(SpanKind.Client)
            .build
            .resource

          headers  <- Resource.eval(Tracer[IO].propagate(Headers.empty))
          response <- poll(client.run(request.withHeaders(headers)).mapK(spanOps.trace))

          _ <- Resource.eval(
            spanOps.span.addAttribute(HttpAttributes.HttpResponseStatusCode(response.status.code))
          )
        } yield response
      }
    }

  private given TextMapUpdater[Headers] with {
    def updated(carrier: Headers, key: String, value: String): Headers =
      carrier.put(key -> value)
  }

}

7.3. Wiring everything together

The application entry point assembles modules and owns the lifecycle of the tracing infrastructure. No other part of the codebase needs to know how tracing is configured or how to shut it down.

import cats.effect.{IO, IOApp, Resource}
import org.typelevel.otel4s.oteljava.OtelJava
import org.typelevel.otel4s.trace.*

object Main extends IOApp.Simple {
  def run: IO[Unit] = {
    for {
      otel4s                   <- OtelJava.autoConfigured[IO]()
      given TracerProvider[IO] <- Resource.pure(otel4s.tracerProvider)
      httpClient               <- HttpClient.create
      forecastModule           <- Resource.eval(ForecastModule.create(httpClient))
      httpServer               <- Resource.eval(HttpServer.create(forecastModule))
      _                        <- httpServer.start
    } yield ()
  }.useForever
}

Tracing initializes once with OtelJava.autoConfigured. All modules share a TracerProvider. Each module derives a Tracer with a unique instrumentation scope. Configuration and lifecycle remain at the application edge and infrastructure does not leak into domain code.

7.4. Configuration

All observability configuration is handled externally. The code does not hardcode exporters, endpoints, or sampling policies. These are provided through environment variables and system properties.

This allows the same service to run in different environments with different observability setups. Traces remain structurally identical, but their resource attributes reflect where and how they were produced.

At a minimum, the application defines a service name and enables SDK autoconfiguration.

javaOptions += "-Dotel.service.name=weather-service"
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true"

//> using javaOpt "-Dotel.service.name=weather-service"
//> using javaOpt "-Dotel.java.global-autoconfigure.enabled=true"

7.5. Seeing traces in Grafana

To validate the traces, we run the service locally, send a real request, and inspect the resulting trace in Grafana using a local OpenTelemetry stack.

For local development, Grafana LGTM is a convenient option. It provides a setup with Grafana for data visualization, Tempo for distributed tracing, and an OpenTelemetry Collector, all in a single Docker container. This keeps the feedback loop short and avoids any custom infrastructure work.

We can run it as a Docker container:

$ docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

Once the container is running, the OpenTelemetry Collector is ready to receive traces, and Grafana is available to query them.

Next, we start the application itself:

$ sbt run

$ scala-cli run Main.scala

With both running, we send a request to the service:

$ curl localhost:8080/weather/Kyiv
Kyiv: ⛅️  -12°C

From the caller's perspective, this is an ordinary HTTP request. Behind the scenes, each request produces a trace made up of spans that mirror the execution path. Server spans wrap request handling, client spans wrap outbound calls, and internal spans reflect meaningful application steps.

The application logs confirm that the tracing context was propagated to the outbound request. The injected traceparent header contains this tracing information and demonstrates successful propagation:

11:07:11.521 [io-compute-6] INFO org.http4s.client.middleware.RequestLogger -- 
  HTTP/1.1 GET https://wttr.in/Kyiv?format=3 
  Headers(traceparent: 00-0ceda30d45b3e923b54a853df0762c64-70a2f1612420ee06-01) 
  body=""

Now let's open Grafana and inspect the exported trace:

Tracer[IO]
  .spanBuilder(s"${req.method} ${req.uri}")
  .withSpanKind(SpanKind.Server)
  .build

javaOptions += "-Dotel.service.name=weather-service"

Tracer[IO]
  .spanBuilder(s"${req.method} ${req.uri}")
  .withSpanKind(SpanKind.Server)
  .build

TracerProvider[IO].get("org.http4s.server")

span.addAttribute(
  HttpAttributes.HttpResponseStatusCode(response.status.code)
)

Tracer[IO].span(
  "checkWeather", 
  Attribute("location", location)
)

TracerProvider[IO]
  .tracer("module.forecast")
  .withVersion("1.2.3")
  .get

TracerProvider[IO]
  .tracer("module.forecast")
  .withVersion("1.2.3")
  .get

Tracer[IO].span(
  "checkWeather", 
  Attribute("location", location)
)

span.addEvent("forecast-received")

span.setStatus(StatusCode.Ok)

Tracer[IO]
  .spanBuilder(s"${req.method} ${req.uri}")
  .withSpanKind(SpanKind.Client)
  .build
  .resource

Tracer[IO]
  .spanBuilder(s"${req.method} ${req.uri}")
  .withSpanKind(SpanKind.Client)
  .build
  .resource

TracerProvider[IO].get("org.http4s.client")

The trace has a clear shape. At the top is the server span, which represents the middleware processing of the incoming HTTP request and covers the full request lifecycle. Nested beneath are spans from the business logic layer, each detailing the forecast operation and its internal steps. The outbound HTTP call appears as a client span, indicating an external request during the trace.

Span names indicate the operations they describe. Parent-child relationships, where one operation triggers another, follow actual execution. Timing bars show when each operation ran and for how long. HTTP spans represent network requests, while domain spans relate to specific business logic, making them easy to distinguish. Instrumentation scopes, or labeled code sections, clarify which application layer generated each span.

Attributes at the HTTP layer, such as http.response.status_code=200, appear only on transport spans and indicate the request's outcome. Domain attributes, specific to business logic, appear only on domain spans. An attribute like location=Kyiv helps categorize the span by business context. Events indicate significant actions or state changes and appear as markers within spans.

Even in this small example, the trace already answers practical questions. You see where time was spent and which part of the request performed the work. You also see how different operations were nested. As the application grows, this structure scales. Adding more routes increases server spans. More domain logic adds business spans. The overall shape of the trace stays stable as it grows.

8. Conclusion

Distributed tracing with otel4s works best when it is built into the program rather than an afterthought layered on top. The API encourages clear ownership and disciplined context flow, which matches how real systems behave under load.

In this article, we looked at how traces are identified, how spans are scoped and propagated, and how otel4s translates OpenTelemetry concepts onto functional code.

The practical takeaway is simple. Instrument the operations that explain system behavior, keep spans aligned with meaningful work, and let context propagation do the rest. When tracing is consistent, the resulting traces align with the work and give you answers rather than more noise.

In the next part of the series, we will apply the same mindset to metrics and examine how to measure system behavior over time using otel4s.