Logo

dev-resources.site

for different kinds of informations.

A protoc compiler plugin that generates useful extension code for Kotlin/JVM

Published at
5/29/2024
Categories
kotlin
protobuf
Author
behase
Categories
2 categories in total
kotlin
open
protobuf
open
Author
6 person written this
behase
open
A protoc compiler plugin that generates useful extension code for Kotlin/JVM

TL;DR

https://github.com/be-hase/protoc-gen-kotlin-ext

  • Generate *OrNull extension properties for optional field
  • Generate factory functions that resemble data class constructors
  • We do not generate our own code, so we can take advantage of the official ecosystem
    • It only generates additional useful code.

Example

Let's assume we have the following proto file.

message Person {
  string first_name = 1;
  string last_name = 2;
  optional string middle_name = 3;
  Gender gender = 4;
  optional string nickname = 5;
  Address primary_address = 6;
}

enum Gender {
  GENDER_UNSPECIFIED = 0;
  MALE = 1;
  FEMALE = 2;
  OTHERS = 3;
}

message Address {
  string country = 1;
  string state = 2;
  string city = 3;
  string address_line_1 = 4;
  optional string address_line_2 = 5;
}
Enter fullscreen mode Exit fullscreen mode

Using the generated code, you can write the following:

fun main() {
    // You can use factory functions that resemble data class constructors
    val person = Person(
        firstName = "Ryosuke",
        lastName = "Hasebe",
        middleName = null, // Optional fields become nullable
        gender = Gender.MALE,
        nickname = null, // Optional fields become nullable
        primaryAddress = Address(
            country = "JP",
            state = "Tokyo",
            city = "blah blah blah",
            addressLine1 = "blah blah blah",
            addressLine2 = null, // Optional fields become nullable
        ),
    )

    // You can use `*OrNull` extension properties for optional field
    println(person.middleNameOrNull)
    println(person.nicknameOrNull)
    println(person.primaryAddress.addressLine2OrNull)
    println(person.primaryAddressOrNull)
}
Enter fullscreen mode Exit fullscreen mode

Motivation

Challenges with Pptional Field (Field Presence)

From protobuf 3.12, the long-awaited optional field (Field Presence) has been reintroduced. This allows the
representation of null (as in java/kotlin).

message Sample {
  optional string hoge = 2;
  optional int32 bar = 3;
}
Enter fullscreen mode Exit fullscreen mode

However, this optional feature is somewhat tricky. When you retrieve a value with a getter, it returns the default
value (e.g., "" for strings, 0 for int32). This means that for strings, you cannot distinguish whether the value has
been explicitly set to "" or not set at all. Instead, optional fields provide a has* method to check if the value
has been set, which you should use to determine if it is null.

If you are unaware of this specification, you might mistakenly treat an "" or 0 as a valid value, potentially
causing bugs.

Sample.newBuilder().build().also {
    println("[1] hoge: ${it.hoge}")
    println("[1] hasHoge: ${it.hasHoge()}")
    println("[1] bar: ${it.bar}")
    println("[1] hasBar: ${it.hasBar()}")
}
Sample.newBuilder().setHoge("hoge").setBar(1).build().also {
    println("[2] hoge: ${it.hoge}")
    println("[2] hasHoge: ${it.hasHoge()}")
    println("[2] bar: ${it.bar}")
    println("[2] hasBar: ${it.hasBar()}")
}
Enter fullscreen mode Exit fullscreen mode
[1] hoge:
[1] hasHoge: false
[1] bar: 0
[1] hasBar: false
[2] hoge: hoge
[2] hasHoge: true
[2] bar: 1
[2] hasBar: true
Enter fullscreen mode Exit fullscreen mode

Even knowing this specification correctly would result in a large amount of boilerplate code, such as the following:

// 😭
val hoge = if (sample.hasHoge()) {
    sample.hoge
} else {
    null
}
Enter fullscreen mode Exit fullscreen mode

To address this for Kotlin, we will auto-generate *OrNull extension properties. When implementing,
seeing *OrNull
through autocompletion should help avoid some issues. Additionally, this allows the use of the Elvis operator, resulting
in smoother code.

val BlahBlah.hogeOrNull: kotlin.String?
    get() = if (hasHoge()) hoge else null
Enter fullscreen mode Exit fullscreen mode

Interestingly, for message types, using protoc-gen-kotlin already generates similar extension properties.
https://github.com/protocolbuffers/protobuf/blob/b30f3de12f946b6d610c21bc605726bf56ea889f/src/google/protobuf/compiler/java/message.cc#L1352C33-L1370

I have raised an issue requesting the addition of optional
scalar types, but it is not planned to be supported by protoc-gen-kotlin.

Challenges with Java Builder

Protobuf uses builders to set values. Since this code is written in Java, there is no non-null/nullable type checking.
When used from Kotlin, it is often misunderstood that it is okay to set null for optional fields, which results in
passing null and encountering NPEs (NullPointerExceptions).

Sample.newBuilder()
    .setHoge(null) // This code raises NPE 😭
    .setBar(1)
    .build();
Enter fullscreen mode Exit fullscreen mode

When using protoc-gen-kotlin, the following DSL is generated for Kotlin. This is beneficial because it includes
non-null/nullable type checking. Great!

sample {
    hoge = null // This code results in a compile error 😀
    bar = 1
}
Enter fullscreen mode Exit fullscreen mode

However, in Kotlin, it feels more natural to write using constructors with named arguments, like data classes.

Additionally, just like the builder, the DSL does not result in a compile error when a field is added later. This can
lead to cases where values are forgotten to be set. However, opinions may differ on whether this is seen as a
disadvantage or an advantage.

If it is code provided by libraries or similar, it is better not to do it as it may break
compatibility.

If it is code used within your own application, I think it is convenient. If you prioritize a more robust programming
style, it is preferable to have a compile error indicating that a value has been forgotten.

Therefore, we will auto-generate the following factory function. Note that this Sample(...) is not a constructor but a
function.
It can be written similarly to a data class, and when a field is added later, it will result in a compile error,
indicating that a value might be forgotten to be set. Since it is defined in Kotlin, nullable type checking is also
enabled.

// similar to a data class 😀
Sample(
    hoge = "hoge",
    bar = 1
)

// Of course, when it is optional, it becomes a nullable type
Sample(
    hoge = null, // This code results in a compile error 😀
    bar = 1
)

Enter fullscreen mode Exit fullscreen mode
public fun Sample(hoge: String?, bar: Int?): Sample {
    val _builder = Sample.newBuilder().apply {
        hoge?.let { setHoge(it) }
        bar?.let { setBar(it) }
    }
    return _builder.build()
}
Enter fullscreen mode Exit fullscreen mode

We considered adding factory functions with default arguments, but we decided not to create them at this time. If there
are default arguments, adding a field later would not result in a compile error.
If you want to use default values, you can use java builder or kotlin DSL.

How to use?

Gradle

When used with protoc-gen-kotlin:

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:<version>"
    }
    plugins {
        id("kotlin-ext") {
            artifact = "dev.hsbrysk:protoc-gen-kotlin-ext:<version>:jdk8@jar"
        }
    }
    generateProtoTasks {
        all().forEach { task ->
            task.plugins {
                id("kotlin-ext") {
                    outputSubDir = "kotlin"
                }
            }
            task.builtins {
                id("kotlin")
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

When used standalone:

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:<version>"
    }
    plugins {
        id("kotlin-ext") {
            artifact = "dev.hsbrysk:protoc-gen-kotlin-ext:<version>:jdk8@jar"
        }
    }
    generateProtoTasks {
        all().forEach { task ->
            task.plugins {
                id("kotlin-ext") {
                    outputSubDir = "kotlin"
                    // This option is explained later in the document
                    option("messageOrNullGetter+")
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Maven or Manual protoc Usage

Since the plugin is created using the same mechanism as protoc-gen-grpc-kotlin, please refer to this document.

Advanced

Compile Options

Perhaps you only want the *OrNull extension property and do not need the factory function. (and vice versa).

This can be achieved by setting the compile options.

// Here, we will use Gradle as an example.
protobuf {
    // ...
    generateProtoTasks {
        all().forEach { task ->
            task.plugins {
                id("kotlin-ext") {
                    // ...
                    // HERE
                    option("messageOrNullGetter+")
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The following options are available. Appending + to the end of an option enables it, while appending - disables it.

  • factory
    • Whether to generate a factory (default: on)
  • orNullGetter
    • Whether to generate orNull extension functions for optional scalar fields (default: on)
  • messageOrNullGetter
    • Whether to generate orNull extension functions for optional message fields (default: off)
    • Not needed when using protobuf-kotlin.

When specifying multiple options, please separate them with commas. For
example, factory-, orNullGetter+, messageOrNullGetter+

protobuf Article's
30 articles in total
Favicon
Protocol Buffers as a Serialization Format
Favicon
Part 2: Defining the Authentication gRPC Interface
Favicon
Compile Protocol Buffers & gRPC to Typescript with Yarn
Favicon
Use RBAC to protect your gRPC service right on proto definition
Favicon
Gamechanger Protobuf
Favicon
Gamechanger Protobuf
Favicon
RPC Action EP2: Using Protobuf and Creating a Custom Plugin
Favicon
FauxRPC
Favicon
Why should we use Protobuf in Web API as data transfer protocol.
Favicon
JSON vs FlatBuffers vs Protocol Buffers
Favicon
gRPC - Unimplemented Error 12
Favicon
A protoc compiler plugin that generates useful extension code for Kotlin/JVM
Favicon
Reducing flyxc data usage
Favicon
Koinos, Smart Contracts, WASM & Protobuf
Favicon
This Week I Learnt: gRPC & Protobuf
Favicon
Building a gRPC Server with NestJS and Buf: A Comprehensive Showcase
Favicon
Exploring Alternatives: Are There Better Options Than JSON?
Favicon
Creating the Local First Stack
Favicon
Roll your own auth with Rust and Protobuf
Favicon
OCaml, Python and protobuf
Favicon
Introduction to Protocol Buffers
Favicon
Using Protobuf with TypeScript
Favicon
[Typia] I made Protocol Buffer library of TypeScript, easiest in the world
Favicon
Protoc Plugins with Go
Favicon
Using Azure Web PubSub with Protobuf subprotocol in .NET
Favicon
A secret weapon to improve the efficiency of golang development, a community backend service was developed in one day
Favicon
Linting Proto Files With Buf
Favicon
What is gRPC
Favicon
fast framework for binary serialization and deserialization in Java, and has the fewest serialization bytes
Favicon
Protobuf vs Avro for Kafka, what to choose?

Featured ones: