Generated JSON-Serialisation for Kotlin

For quite some time I used Gson for serialisation in our Kotlin and Java-applications. It was the most brittle part of our code, for several reasons:

• When Gson doesn’t find a no-arg constructor, sun.misc.Unsafe is used. This means initialisers aren’t called and the deserialised objects could be broken.
• Gson uses reflection to set and retrieve fields. Reflection is slow on Android.
• Nulls in JSON are able to break Kotlin’s null-safe typing.

To improve on that situation, I decided to write an Annotation Processor that generates serialisation code at compile-time, for that would give the most freedom. Here are my design-choices:

All necessary data needs to be in the constructor

I decided to instantiate deserialised objects with an existing constructor only (the one with the most arguments). This ensures that they are always initialised properly.

The only needed annotation is @GsonSerializer which goes on the top of a class:

@GsonSerializer
class AppState(val userName: String,
               val notifications: List<String>)

This forces separation of initial from generated data. It also removes the need to exclude lazy or @Transient properties.

The generated serialiser is an extension function on JsonWriter:

fun JsonWriter.value(obj: AppState?) {
    if (obj == null) {
        nullValue()
        return
    }
    beginObject()
    name("userName").value(obj.userName)
    name("notifications").array(obj.notifications) { value(it) }
    endObject()
}

The field names and types are taken from the constructor. The code assumes they are also accessible as members, which is easy to obtain.

Developer-friendly deserialisation

Deserialisation is more complex, since it does a couple of things to help development:

• It throws informative errors when fields are missing
• Field-names and enum-values are matched case-insensitive. I realise this is debatable, but it prevents some flow-breaking bugs and was never an issue in production. Serialisation keeps the original casing.

The deserialiser:

fun JsonReader.nextAppState(): AppState {

    var userName: String? = null
    var notifications: MutableList<String>? = null

    beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "username" -> userName = nextString()
            "notifications" -> notifications = readList(JsonReader::nextString)
            else -> skipValue()
        }
    }
    endObject()

    val obj = AppState(
        userName = checkNotNull(userName) {"error parsing AppState, field not found: userName: String"},
        notifications = checkNotNull(notifications) {"error parsing AppState, field not found: notifications: List"}
    )

    return obj
}

Nested objects

Since all methods are created as extensions on JsonWriter and JsonReader adding support for more fields is as simple as creating a new extension method or adding @GsonSerializer to the referenced class:

@GsonSerializer
class AppState(val user: UserInfo,
               val time: ZonedDateTime)

class UserInfo(name: String)

The generated code for this example won’t compile because it misses:

• JsonWriter.value(UserInfo)
• JsonReader.nextUserInfo()
• JsonWriter.value(ZonedDateTime)
• JsonReader.nextZonedDateTime()

To make it work add @GsonSerializer to UserInfo and add the following code for ZonedDateTime

fun JsonWriter.value(obj: ZonedDateTime) {
    value(obj.toString())
}

fun JsonReader.nextZonedDateTime(): ZonedDateTime {
    return ZonedDateTime.parse(nextString())
}

Abstract Types

Reading and writing abstract types is necessary and possible, consider this:

@GsonSerializer
class AppState(val requests: Set<NetworkRequest>)

@GsonSerializer
interface NetworkRequest

@GsonSerializer
data class LoginReq(val username: String, val password: String) : NetworkRequest

The processor checks the type-hierarchy and writes the object and type-name into an array. I chose an array over an object so I wouldn’t have to cache anything in case the type-name arrives after the data during deserialisation.

fun JsonWriter.abstractValue(obj: NetworkRequest?) {
    if (obj == null) {
        nullValue()
        return
    }
    beginArray()
    when (obj) {
        is LoginReq -> value("LoginReq").value(obj)
        else -> throw IllegalStateException("type $obj not annotated")
    }
    endArray()
}

/* deserialize */
fun JsonReader.nextNetworkRequest(): NetworkRequest {
    beginArray()
    val typeName = nextString()
    val obj: NetworkRequest = when (typeName) {
        "LoginReq" -> nextLoginReq()
        else -> throw IllegalStateException("unknown type: $typeName")
    }
    endArray()

    return obj
}

Graphs and circular objects

Using another annotation @ParentRef, the processor is able to process circular graphs, like a leaf-node having a references to it’s root

@GsonSerializer
class Root(var leaf: Leaf?)

@GsonSerializer
data class Leaf(@ParentRef val root: Root, val number: Int)

The deserialiser for Leaf returns a function now:

fun JsonReader.nextLeaf(): (root: Root) -> Leaf {
    var number: Int? = null

    beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "number" -> number = nextInt()
            else -> skipValue()
        }
    }
    endObject()

    return { root->
        Leaf(
            root = root,
            number = checkNotNull(number) {"error parsing Leaf, field not found: number: Int"}
        )
    }
}

For type-safe deserialisation to work, the field in the parent-object must be either nullable or a collection-type:

fun JsonReader.nextRoot(): Root {

    var leaf: ((Root) -> Leaf)? = null
    
    beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "leaf" ->  {
                 leaf = nextOrNull(JsonReader::nextLeaf)
             }
            else -> skipValue()
        }
    }
    endObject()

    /* construct parent */
    val obj = Root(
        leaf = null
    )

    /* set child */
    obj.leaf = leaf?.invoke(obj)

    return obj
}

Summary

This was a short overview about my new method of serialisation. I can’t share any code yet but the principles are clear.

The choice of compile-time-generation and Kotlin with it’s extensions methods and lambdas makes it possible to easily create convenient and fast serialisation-code.

Assuming all necessary data to be passed in the constructor helps keeping everything safe and enforces separation of initial from generated data.

I hope you can take some ideas with you. Happy coding.