Distributed Tracing

Mu provides an integration with Natchez to enable distributed tracing of gRPC calls.

Example of a distributed trace

Specifically, the integration provides the following features.

Client

  • For every RPC call, the client will create a child span with the fully qualified name of the RPC method being called (e.g. com.foo.Greeter/SayHello)
  • It will automatically add all necessary trace-related headers to RPC requests.

Server

  • The server will attempt to extract trace-related information from the request headers.
  • It will create a span using the same naming convention as the client.
    • If the relevant headers were present, it will continue the trace that was started upstream, creating a child span.
    • Otherwise, it will create a root span, i.e. a new trace.

How to use

Please, be sure you’ve checked the Accessing metadata on services first.

Let’s look at how to enable tracing on the server side first.

Server side

We’ll assume the following service definition:

case class HelloRequest(name: String)
case class HelloResponse(greeting: String, happy: Boolean)

trait Greeter[F[_]] {

  def SayHello(req: HelloRequest): F[HelloResponse]

}

and an implementation of that definition:

import mu.examples.protobuf.greeter.*
import cats.Applicative
import cats.syntax.applicative.*

class MyAmazingGreeter[F[_]: Applicative] extends Greeter[F] {

  def SayHello(req: HelloRequest): F[HelloResponse] =
    HelloResponse(s"Hello, ${req.name}!", happy = true).pure[F]

}

To use this service with tracing enabled, you need to call the Greeter.bindContextService[F, Span[F]] method instead of the usual Greeter.bindService[F].

That requires an instance of ServerContext[F, Span[F]], which is available in the higherkindness.mu.rpc.internal.tracing.implicits object:

import higherkindness.mu.rpc.internal.context.ServerContext
import natchez.{EntryPoint, Span}

implicit def serverContext[F[_]](implicit entrypoint: EntryPoint[F]): ServerContext[F, Span[F]]

So, to trace our service, we need to call to Greeter.bindContextService[F, Span[F]] with the import higherkindness.mu.rpc.internal.tracing.implicits.* in the scope and providing a Natchez EntryPoint implicitly.

EntryPoint

EntryPoint[F[_]], as the name suggests, is the “entrypoint” into the Natchez API. It’s what allows Mu to do things like create root spans.

How you create an EntryPoint will depend on what tracing implementation you want to use. For example, if you use natchez-jaeger, you might create a Resource of an EntryPoint like this:

import cats.effect.{Sync, Resource}

import natchez.EntryPoint
import natchez.jaeger.Jaeger
import io.jaegertracing.Configuration.SamplerConfiguration
import io.jaegertracing.Configuration.ReporterConfiguration

def entryPoint[F[_]: Sync]: Resource[F, EntryPoint[F]] = {
  Jaeger.entryPoint[F]("my-Mu-service") { c =>
    Sync[F].delay {
      c.withSampler(SamplerConfiguration.fromEnv)
       .withReporter(ReporterConfiguration.fromEnv)
       .getTracer
    }
  }
}

Kleisli

When you instantiate your Greeter implementation, you need to set its type parameter to Kleisli[F, Span[F], *]. (Note: we are using kind-projector syntax here, but you don’t have to.)

Intuitively, this creates a service which, given the current span as input, returns a result inside the F effect.

Luckily, there are instances of most of the cats-effect type classes for Kleisli, all the way down to Async. So you should be able to substitute Greeter[Kleisli[F, Span[F], *]] for Greeter[F] without requiring any changes to your service implementation code.

Using bindContextService

Putting all this together, your server setup code will look something like this:

import cats.effect.*
import cats.data.Kleisli
import higherkindness.mu.rpc.server.*
import natchez.Span

object TracingServer extends IOApp {

  import higherkindness.mu.rpc.internal.tracing.implicits.*

  given Greeter[Kleisli[IO, Span[IO], *]] =
    new MyAmazingGreeter[Kleisli[IO, Span[IO], *]]

  def run(args: List[String]): IO[ExitCode] =
    entryPoint[IO]
      .flatMap { implicit ep =>
        Greeter.bindContextService[IO, Span[IO]]
      }
      .flatMap { serviceDef =>
        GrpcServer.defaultServer[IO](8080, List(AddService(serviceDef)))
      }.useForever

}

Tracing your service code

If you wish, you can make use of the Natchez Trace typeclass to create child spans:

import natchez.Trace
import cats.Monad
import cats.syntax.all.*

class MyTracingService[F[_]: Monad: Trace] extends Greeter[F] {

  def SayHello(req: HelloRequest): F[HelloResponse] =
    for {
      _ <- Trace[F].span("look stuff up in the database"){ Monad[F].unit }
      _ <- Trace[F].span("do some stuff with Redis"){ Monad[F].unit }
      _ <- Trace[F].span("make an HTTP call"){ Monad[F].unit }
    } yield HelloResponse(s"Hi, ${req.name}!")

}

Client side

To obtain a tracing client, use Greeter.contextClient[F, Span[F]] instead of Greeter.client.

This returns a Greeter[Kleisli[F, Span[F], *]], i.e. a client which takes the current span as input and returns a response inside the F effect.

Similar to the server side, there’s a ClientContext[F, Span[F]] instance available in the higherkindness.mu.rpc.internal.tracing.implicits object:

import cats.effect.Async
import higherkindness.mu.rpc.internal.context.ClientContext
import natchez.Span

implicit def clientContext[F[_]: Async]: ClientContext[F, Span[F]]

For example:

import higherkindness.mu.rpc.*

object TracingClientApp extends IOApp {

  import higherkindness.mu.rpc.internal.tracing.implicits.*

  val channelFor: ChannelFor = ChannelForAddress("localhost", 8080)

  val clientRes: Resource[IO, Greeter[Kleisli[IO, Span[IO], *]]] =
    Greeter.contextClient[IO, Span[IO]](channelFor)

  def run(args: List[String]): IO[ExitCode] =
    entryPoint[IO].use { ep =>
      ep.root("this is the root span").use { currentSpan =>
        clientRes.use { client =>
          val kleisli = client.SayHello(HelloRequest("Chris"))
          for {
            resp <- kleisli.run(currentSpan)
            _    <- IO(println(s"Response: $resp"))
          } yield (ExitCode.Success)
        }
      }
    }

}

Working example

To see a full working example of distributed tracing across multiple Mu services, take a look at this project in the mu-scala-examples repo: higherkindness/mu-scala-examples.

The README explains how to run the example and inspect the resulting traces.