Generating sources from Avro

Getting started

First add the sbt plugin in project/plugins.sbt:

addSbtPlugin("io.higherkindness" % "sbt-mu-srcgen" % "0.31.1")

And enable the plugin on the appropriate project(s):

enablePlugins(SrcGenPlugin)

Once the plugin is enabled, you can configure it by adding a few lines to build.sbt:

import higherkindness.mu.rpc.srcgen.Model._

// Look for Avro IDL files
muSrcGenIdlType := IdlType.Avro

Suppose you want to generate Scala code for a gRPC service based on the following Avro IDL file, src/main/resources/hello.avdl:

@namespace("foo")
protocol AvroGreeter {

    record HelloRequest {
        string arg1;
        union { null, string } arg2;
        array<string> arg3;
    }

    record HelloResponse {
        string arg1;
        union { null, string } arg2;
        array<string>  arg3;
    }

    foo.HelloResponse sayHelloAvro(foo.HelloRequest arg);
}

NOTE: please be aware that mu-scala restricts Avro RPC method arguments to a single record type and only permits records as return types; for more context, see the source generation reference.

You can run the source generator directly:

sbt muSrcGen

or as part of compilation:

sbt compile

Once the source generator has run, there should be a generated Scala file at target/scala-3.1.2/src_managed/main/foo/AvroGreeter.scala.

It will look like this (tidied up and simplified for readability):

package foo

final case class HelloRequest(
  arg1: String,
  arg2: Option[String],
  arg3: Seq[String]
)

final case class HelloResponse(
  arg1: String,
  arg2: Option[String],
  arg3: Seq[String]
)

trait AvroGreeter[F[_]] {
  def sayHelloAvro(arg: HelloRequest): F[HelloResponse]
}

object AvroGreeter {

  // ... lots of generated code

}

It’s also possible to generate Scala code from .avpr (JSON) files.

Suppose you delete src/main/resources/hello.avdl and replace it with src/main/resources/hello.avpr:

{
  "namespace" : "foo",
  "protocol" : "AvroGreeter",
  "types" : [
    {
      "name" : "HelloRequest",
      "type" : "record",
      "fields" : [
        {
          "name" : "arg1",
          "type" : "string"
        },
        {
          "name" : "arg2",
          "type" : [
            "null",
            "string"
          ]
        },
        {
          "name" : "arg3",
          "type" : {
            "type" : "array",
            "items" : "string"
          }
        }
      ]
    },
    {
      "name" : "HelloResponse",
      "type" : "record",
      "fields" : [
        {
          "name" : "arg1",
          "type" : "string"
        },
        {
          "name" : "arg2",
          "type" : [
            "null",
            "string"
          ]
        },
        {
          "name" : "arg3",
          "type" : {
            "type" : "array",
            "items" : "string"
          }
        }
      ]
    }
  ],
  "messages" : {
    "sayHelloAvro" : {
      "request" : [
        {
          "name" : "arg",
          "type" : "HelloRequest"
        }
      ],
      "response" : "HelloResponse"
    }
  }
}

If you run sbt clean muSrcGen, you should end up with exactly the same generated Scala file as before.

Avro code generation details

This section explains the different Scala structures that are generated from Avro IDL.

To achieve this generation Mu’s source generator uses avrohugger behind the scenes.

Avro Protocols

Let’s start from the beginning, everything in Avro IDL should be declared inside a protocol.

The name of that protocol will be the name of our Scala file.

protocol People {
 ...
}

muSrcGen =>

People.scala

Furthermore, the protocol can have a namespace which will be our Scala package:

@namespace("example.protocol")
protocol People {
 ...
}

muSrcGen =>

example/protocol/People.scala

Messages

In Avro IDL, the messages are declared with the keyword record and contain fields. The record will be translated to a case class with corresponding fields:

record Person {
  string name;
  int age;
  boolean crossfitter;
}

muSrcGen =>

case class Person(name: String, age: Int, crossfitter: Boolean)

Enums

Avro supports enums too and they are translated to a Scala sealed trait + case objects:

enum Errors {
  NotFound, Duplicated, None
}

muSrcGen =>

sealed trait Errors

object Errors {
  @AvroSortPriority(0) case object NotFound extends Errors
  @AvroSortPriority(1) case object Duplicated extends Errors
  @AvroSortPriority(2) case object None extends Errors
}

Unions

Unions are a complex Avro type for fields inside records. As its name suggest, it represents a type composed by another types.

Depending on the types composing the union, Mu will interpret it on different ways:

Optional fields

When we add a null to a union expression, we’ll get a Scala Option of the other type declared along the null:

record PeopleRequest {
  union {null, string} name;
}

muSrcGen =>

case class PeopleRequest(name: Option[String])

Eithers

When we join two non-null types on a union we’ll get an Scala Either with the same types order:

record PeopleResponse {
  union { Errors, Person } result;
}

muSrcGen =>

case class PeopleResponse(result: Either[Errors.Value, Person])

Scala union types

And finally, when we have three or more non-null types on a single union, a TaggedUnion will be used. This is a wrapper around a Scala 3 union type, defined by Mu.

record PeopleResponse {
  union{ string, int, Errors } result;
}

muSrcGen =>

case class PeopleResponse(result: TaggedUnion3[String, Int, Errors])

Services

When we declare a method or endpoint inside a protocol this will be converted to a trait to define a Mu service.

If we want to keep our models separate from our service definitions, Avro allows us to import other Avro files and use their records:

protocol PeopleService {
  import idl "People.avdl"; //Under the same folder

  example.protocol.PeopleResponse getPerson(example.protocol.PeopleRequest request);
}

muSrcGen =>

trait PeopleService[F[_]] {
  def getPerson(request: example.protocol.PeopleRequest): F[example.protocol.PeopleResponse]
}

Also, an endpoint can be declared without params or not returning anything. Mu will use its Empty type to cover these cases:

protocol PeopleService {
  void insertPerson();
}

muSrcGen =>

trait PeopleService[F[_]] {
  def insertPerson(arg: Empty.type): F[Empty.type]
}

For a full understanding of the Avro syntax we recommend you to take a look to the Avro Official site where you can find all the Avro supported types and some interesting resources.