How to validate that a Protobuf message does not contain enum fields with zero value? Turn out that is not supported directly by Protobuf! We need to look into how protojson package is implemented. More and more companies are adopting gRPC with for communication between internal services. It has the benefits of high performance, supporting multiple programming languages, and being backed by Google with a great ecosystem around. Protobuf For communication with front-end and external services, Protobuf can be marshaled to JSON format. The browser only understands JSON format, and we can not expect other companies to consume Protobuf directly from us. (Of course, you can, if you are !) big enough Let’s talk about the enum and some problems that we encountered recently with it. Sample code is written in Go. From the Protobuf , the zero value enum should have the suffix . It’s because enum is implemented as a , and the value is considered as, well, unspecified. It’s similar to for a message or an empty string. When encoding Protobuf as JSON, a message, an enum, or an empty string is ignored. style guide UNSPECIFIED uint32 0 nil nil UNSPECIFIED We were following that convention, until someday, we did not. When sending external webhook messages, we decided to not use as . One reason is that we are using to ensure that all fields are included in the JSON representation when sending webhook messages to external parties. value to appear in the webhook messages, if somehow we forget to set an enum field to 0. Unit tests can not catch all the mistakes; we engineers know that. 0 UNSPECIFIED EmitUnpopulated: true And we don’t want that UNSPECIFIED This causes a lot of trouble, so we had to revert and make the value as again. One problem is that it forces the use of everywhere! And there are places where we don’t want to emit all unpopulated fields. Like calling some third-party APIs. Some messages mix between enums and non- enums; there are no ways to send the correct format with that. Use , the third-party APIs don’t understand ; use and some required fields with non- enums are omitted. Of course, they can all be refactored away, but it should be simpler to just force the use of at the beginning. 0 UNSPECIFIED EmitUnpopulated: true UNSPECIFIED UNSPECIFIED EmitUnpopulated: true UNSPECIFIED EmitUnpopulated: false UNSPECIFIED UNSPECIFIED Well, why not just verify that all enum fields are not set to 0 in webhook messages? You may ask. Turn out there are no simple ways to do that in Protobuf 3! In Protobuf 2, there is option to prevent a field to be unset. This option was removed in Protobuf 3, because it prevents refactoring for removing fields. If we forgot to update every service to remove that no-longer-used field, especially in a company with multiple teams working together, the messages will be dropped unintentionally. It should be better not to require it upfront. ( ) required required more In Protobuf 3, there was interface. We can simply implement that interface for all enums to return error upon seeing a zero value. But again, it was removed! As a protocol, we should minimize the customization as much as possible. Otherwise, that customization will have to be implemented and maintained in all different languages across different teams! jsonpb.JSONPBMarshaler So, how to validate zero value in enum fields? We’ll have to reach the reflection package. The interface has method for iterating over every populated field. We can use that method to verify that there are no enum fields with zero… Oh, wait. It only iterates over . So it won’t detect the zero value in enum! protoreflect.Message Range() populated fields But the function can still emit unpopulated fields with option. How does it implement that? Dive into , there is a code snippet for iterating over ( ). Let’s steal it: protojson.Marshal() EmitUnpopulated encoding/protojson unpopulated fields source // unpopulatedFieldRanger wraps a protoreflect.Message and modifies its Range
// method to additionally iterate over unpopulated fields.
type unpopulatedFieldRanger struct{ pref.Message }

func (m unpopulatedFieldRanger) Range(f func(pref.FieldDescriptor, pref.Value) bool) {
	fds := m.Descriptor().Fields()
	for i := 0; i < fds.Len(); i++ {
		fd := fds.Get(i)
		if m.Has(fd) || fd.ContainingOneof() != nil {
			continue // ignore populated fields and fields within a oneofs
		}

		v := m.Get(fd)
		isProto2Scalar := fd.Syntax() == pref.Proto2 && fd.Default().IsValid()
		isSingularMessage := fd.Cardinality() != pref.Repeated && fd.Message() != nil
		if isProto2Scalar || isSingularMessage {
			v = pref.Value{} // use invalid value to emit null
		}
		if !f(fd, v) {
			return
		}
	}
	m.Message.Range(f)
} What the above code does is iterating over additional fields, by looping over . Fields within fields are skipped. Unpopulated singular fields are set as (think of it as in generated JSON) before being sent to the input function. protoreflect.Message.Descriptor().Fields() oneof message invalid null Still a bit more code to write, like implementing a traveling method for iterating over all different Protobuf types: message, array (repeated), dynamic , and of course, enum. But it’s solvable. And I can take a rest now. Struct Thanks for reading! If you have a better way to do that, please let me know by connecting on 👋 Twitter Also published at . my blog

Google

Twitter

Validating Zero Enum Value in the Protocol Buffer

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Experimenting With Sapling, the New Git Client From Meta

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Experimenting With Sapling, the New Git Client From Meta

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps