What Is Go Protobuf? Exploring The New Opaque API

[Protocol Buffers (Protobuf) is Google’s language-neutral data interchange format. See protobuf.dev.] Protocol Buffers (Protobuf) protobuf.dev Back in March 2020, we released the google.golang.org/protobuf module, a major overhaul of the Go Protobuf API. This package introduced first-class support for reflection, a dynamicpb implementation and the protocmp package for easier testing. google.golang.org/protobuf a major overhaul of the Go Protobuf API support for reflection dynamicpb protocmp That release introduced a new protobuf module with a new API. Today, we are releasing an additional API for generated code, meaning the Go code in the .pb.go files created by the protocol compiler (protoc). This blog post explains our motivation for creating a new API and shows you how to use it in your projects. .pb.go protoc To be clear: We are not removing anything. We will continue to support the existing API for generated code, just like we still support the older protobuf module (by wrapping the google.golang.org/protobuf implementation). Go is committed to backwards compatibility and this applies to Go Protobuf, too! google.golang.org/protobuf committed to backwards compatibility Background: the (existing) Open Struct API We now call the existing API the Open Struct API, because generated struct types are open to direct access. In the next section, we will see how it differs from the new Opaque API. To work with protocol buffers, you first create a .proto definition file like this one: .proto edition = "2023"; // successor to proto2 and proto3 package log; message LogEntry { string backend_server = 1; uint32 request_size = 2; string ip_address = 3; } edition = "2023"; // successor to proto2 and proto3 package log; message LogEntry { string backend_server = 1; uint32 request_size = 2; string ip_address = 3; } Then, you run the protocol compiler (protoc) to generate code like the following (in a .pb.go file): run the protocol compiler ( protoc .pb.go package logpb type LogEntry struct { BackendServer *string RequestSize *uint32 IPAddress *string // …internal fields elided… } func (l *LogEntry) GetBackendServer() string { … } func (l *LogEntry) GetRequestSize() uint32 { … } func (l *LogEntry) GetIPAddress() string { … } package logpb type LogEntry struct { BackendServer *string RequestSize *uint32 IPAddress *string // …internal fields elided… } func (l *LogEntry) GetBackendServer() string { … } func (l *LogEntry) GetRequestSize() uint32 { … } func (l *LogEntry) GetIPAddress() string { … } Now you can import the generated logpb package from your Go code and call functions like proto.Marshal to encode logpb.LogEntry messages into protobuf wire format. logpb proto.Marshal logpb.LogEntry You can find more details in the Generated Code API documentation. Generated Code API documentation (Existing) Open Struct API: Field Presence An important aspect of this generated code is how field presence (whether a field is set or not) is modeled. For instance, the above example models presence using pointers, so you could set the BackendServer field to: field presence BackendServer proto.String("zrh01.prod"): the field is set and contains “zrh01.prod” proto.String(""): the field is set (non-nil pointer) but contains an empty value nil pointer: the field is not set proto.String("zrh01.prod"): the field is set and contains “zrh01.prod” proto.String("zrh01.prod") proto.String(""): the field is set (non-nil pointer) but contains an empty value proto.String("") nil nil pointer: the field is not set nil If you are used to generated code not having pointers, you are probably using .proto files that start with syntax = "proto3". The field presence behavior changed over the years: .proto syntax = "proto3" syntax = "proto2" uses explicit presence by default syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default syntax = "proto2" uses explicit presence by default syntax = "proto2" explicit presence syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword syntax = "proto3" implicit presence opting into explicit presence with the optional edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default edition = "2023" successor to both proto2 and proto3 explicit presence explicit presence The new Opaque API We created the new Opaque API to uncouple the Generated Code API from the underlying in-memory representation. The (existing) Open Struct API has no such separation: it allows programs direct access to the protobuf message memory. For example, one could use the flag package to parse command-line flag values into protobuf message fields: Opaque API Generated Code API flag var req logpb.LogEntry flag.StringVar(&req.BackendServer, "backend", os.Getenv("HOST"), "…") flag.Parse() // fills the BackendServer field from -backend flag var req logpb.LogEntry flag.StringVar(&req.BackendServer, "backend", os.Getenv("HOST"), "…") flag.Parse() // fills the BackendServer field from -backend flag The problem with such a tight coupling is that we can never change how we lay out protobuf messages in memory. Lifting this restriction enables many implementation improvements, which we’ll see below. What changes with the new Opaque API? Here is how the generated code from the above example would change: package logpb type LogEntry struct { xxx_hidden_BackendServer *string // no longer exported xxx_hidden_RequestSize uint32 // no longer exported xxx_hidden_IPAddress *string // no longer exported // …internal fields elided… } func (l *LogEntry) GetBackendServer() string { … } func (l *LogEntry) HasBackendServer() bool { … } func (l *LogEntry) SetBackendServer(string) { … } func (l *LogEntry) ClearBackendServer() { … } // … package logpb type LogEntry struct { xxx_hidden_BackendServer *string // no longer exported xxx_hidden_RequestSize uint32 // no longer exported xxx_hidden_IPAddress *string // no longer exported // …internal fields elided… } func (l *LogEntry) GetBackendServer() string { … } func (l *LogEntry) HasBackendServer() bool { … } func (l *LogEntry) SetBackendServer(string) { … } func (l *LogEntry) ClearBackendServer() { … } // … With the Opaque API, the struct fields are hidden and can no longer be directly accessed. Instead, the new accessor methods allow for getting, setting, or clearing a field. Opaque structs use less memory One change we made to the memory layout is to model field presence for elementary fields more efficiently: The (existing) Open Struct API uses pointers, which adds a 64-bit word to the space cost of the field. The Opaque API uses bit fields, which require one bit per field (ignoring padding overhead). The (existing) Open Struct API uses pointers, which adds a 64-bit word to the space cost of the field. The Opaque API uses bit fields, which require one bit per field (ignoring padding overhead). bit fields Using fewer variables and pointers also lowers load on the allocator and on the garbage collector. The performance improvement depends heavily on the shapes of your protocol messages: The change only affects elementary fields like integers, bools, enums, and floats, but not strings, repeated fields, or submessages (because it is less profitable for those types). less profitable Our benchmark results show that messages with few elementary fields exhibit performance that is as good as before, whereas messages with more elementary fields are decoded with significantly fewer allocations: │ Open Struct API │ Opaque API │ │ allocs/op │ allocs/op vs base │ Prod#1 360.3k ± 0% 360.3k ± 0% +0.00% (p=0.002 n=6) Search#1 1413.7k ± 0% 762.3k ± 0% -46.08% (p=0.002 n=6) Search#2 314.8k ± 0% 132.4k ± 0% -57.95% (p=0.002 n=6) │ Open Struct API │ Opaque API │ │ allocs/op │ allocs/op vs base │ Prod#1 360.3k ± 0% 360.3k ± 0% +0.00% (p=0.002 n=6) Search#1 1413.7k ± 0% 762.3k ± 0% -46.08% (p=0.002 n=6) Search#2 314.8k ± 0% 132.4k ± 0% -57.95% (p=0.002 n=6) Reducing allocations also makes decoding protobuf messages more efficient: │ Open Struct API │ Opaque API │ │ user-sec/op │ user-sec/op vs base │ Prod#1 55.55m ± 6% 55.28m ± 4% ~ (p=0.180 n=6) Search#1 324.3m ± 22% 292.0m ± 6% -9.97% (p=0.015 n=6) Search#2 67.53m ± 10% 45.04m ± 8% -33.29% (p=0.002 n=6) │ Open Struct API │ Opaque API │ │ user-sec/op │ user-sec/op vs base │ Prod#1 55.55m ± 6% 55.28m ± 4% ~ (p=0.180 n=6) Search#1 324.3m ± 22% 292.0m ± 6% -9.97% (p=0.015 n=6) Search#2 67.53m ± 10% 45.04m ± 8% -33.29% (p=0.002 n=6) (All measurements done on an AMD Castle Peak Zen 2. Results on ARM and Intel CPUs are similar.) Note: proto3 with implicit presence similarly does not use pointers, so you will not see a performance improvement if you are coming from proto3. If you were using implicit presence for performance reasons, forgoing the convenience of being able to distinguish empty fields from unset ones, then the Opaque API now makes it possible to use explicit presence without a performance penalty. Motivation: Lazy Decoding Lazy decoding is a performance optimization where the contents of a submessage are decoded when first accessed instead of during proto.Unmarshal. Lazy decoding can improve performance by avoiding unnecessarily decoding fields which are never accessed. proto.Unmarshal Lazy decoding can’t be supported safely by the (existing) Open Struct API. While the Open Struct API provides getters, leaving the (un-decoded) struct fields exposed would be extremely error-prone. To ensure that the decoding logic runs immediately before the field is first accessed, we must make the field private and mediate all accesses to it through getter and setter functions. This approach made it possible to implement lazy decoding with the Opaque API. Of course, not every workload will benefit from this optimization, but for those that do benefit, the results can be spectacular: We have seen logs analysis pipelines that discard messages based on a top-level message condition (e.g. whether backend_server is one of the machines running a new Linux kernel version) and can skip decoding deeply nested subtrees of messages. backend_server As an example, here are the results of the micro-benchmark we included, demonstrating how lazy decoding saves over 50% of the work and over 87% of allocations! │ nolazy │ lazy │ │ sec/op │ sec/op vs base │ Unmarshal/lazy-24 6.742µ ± 0% 2.816µ ± 0% -58.23% (p=0.002 n=6) │ nolazy │ lazy │ │ B/op │ B/op vs base │ Unmarshal/lazy-24 3.666Ki ± 0% 1.814Ki ± 0% -50.51% (p=0.002 n=6) │ nolazy │ lazy │ │ allocs/op │ allocs/op vs base │ Unmarshal/lazy-24 64.000 ± 0% 8.000 ± 0% -87.50% (p=0.002 n=6) │ nolazy │ lazy │ │ sec/op │ sec/op vs base │ Unmarshal/lazy-24 6.742µ ± 0% 2.816µ ± 0% -58.23% (p=0.002 n=6) │ nolazy │ lazy │ │ B/op │ B/op vs base │ Unmarshal/lazy-24 3.666Ki ± 0% 1.814Ki ± 0% -50.51% (p=0.002 n=6) │ nolazy │ lazy │ │ allocs/op │ allocs/op vs base │ Unmarshal/lazy-24 64.000 ± 0% 8.000 ± 0% -87.50% (p=0.002 n=6) Motivation: reduce pointer comparison mistakes Modeling field presence with pointers invites pointer-related bugs. Consider an enum, declared within the LogEntry message: LogEntry message LogEntry { enum DeviceType { DESKTOP = 0; MOBILE = 1; VR = 2; }; DeviceType device_type = 1; } message LogEntry { enum DeviceType { DESKTOP = 0; MOBILE = 1; VR = 2; }; DeviceType device_type = 1; } A simple mistake is to compare the device_type enum field like so: device_type if cv.DeviceType == logpb.LogEntry_DESKTOP.Enum() { // incorrect! if cv.DeviceType == logpb.LogEntry_DESKTOP.Enum() { // incorrect! Did you spot the bug? The condition compares the memory address instead of the value. Because the Enum() accessor allocates a new variable on each call, the condition can never be true. The check should have read: Enum() if cv.GetDeviceType() == logpb.LogEntry_DESKTOP { if cv.GetDeviceType() == logpb.LogEntry_DESKTOP { The new Opaque API prevents this mistake: Because fields are hidden, all access must go through the getter. Motivation: reduce accidental sharing mistakes Let’s consider a slightly more involved pointer-related bug. Assume you are trying to stabilize an RPC service that fails under high load. The following part of the request middleware looks correct, but still the entire service goes down whenever just one customer sends a high volume of requests: logEntry.IPAddress = req.IPAddress logEntry.BackendServer = proto.String(hostname) // The redactIP() function redacts IPAddress to 127.0.0.1, // unexpectedly not just in logEntry *but also* in req! go auditlog(redactIP(logEntry)) if quotaExceeded(req) { // BUG: All requests end up here, regardless of their source. return fmt.Errorf("server overloaded") } logEntry.IPAddress = req.IPAddress logEntry.BackendServer = proto.String(hostname) // The redactIP() function redacts IPAddress to 127.0.0.1, // unexpectedly not just in logEntry *but also* in req! go auditlog(redactIP(logEntry)) if quotaExceeded(req) { // BUG: All requests end up here, regardless of their source. return fmt.Errorf("server overloaded") } Did you spot the bug? The first line accidentally copied the pointer (thereby sharing the pointed-to variable between the logEntry and req messages) instead of its value. It should have read: logEntry req logEntry.IPAddress = proto.String(req.GetIPAddress()) logEntry.IPAddress = proto.String(req.GetIPAddress()) The new Opaque API prevents this problem as the setter takes a value (string) instead of a pointer: string logEntry.SetIPAddress(req.GetIPAddress()) logEntry.SetIPAddress(req.GetIPAddress()) Motivation: Fix Sharp Edges: reflection To write code that works not only with a specific message type (e.g. logpb.LogEntry), but with any message type, one needs some kind of reflection. The previous example used a function to redact IP addresses. To work with any type of message, it could have been defined as func redactIP(proto.Message) proto.Message { … }. logpb.LogEntry func redactIP(proto.Message) proto.Message { … } Many years ago, your only option to implement a function like redactIP was to reach for Go’s reflect package, which resulted in very tight coupling: you had only the generator output and had to reverse-engineer what the input protobuf message definition might have looked like. The google.golang.org/protobuf module release (from March 2020) introduced Protobuf reflection, which should always be preferred: Go’s reflect package traverses the data structure’s representation, which should be an implementation detail. Protobuf reflection traverses the logical tree of protocol messages without regard to its representation. redactIP Go’s reflect google.golang.org/protobuf Protobuf reflection reflect Unfortunately, merely providing protobuf reflection is not sufficient and still leaves some sharp edges exposed: In some cases, users might accidentally use Go reflection instead of protobuf reflection. providing For example, encoding a protobuf message with the encoding/json package (which uses Go reflection) was technically possible, but the result is not canonical Protobuf JSON encoding. Use the protojson package instead. encoding/json canonical Protobuf JSON encoding protojson The new Opaque API prevents this problem because the message struct fields are hidden: accidental usage of Go reflection will see an empty message. This is clear enough to steer developers towards protobuf reflection. Motivation: Making the ideal memory layout possible The benchmark results from the More Efficient Memory Representation section have already shown that protobuf performance heavily depends on the specific usage: How are the messages defined? Which fields are set? More Efficient Memory Representation To keep Go Protobuf as fast as possible for everyone, we cannot implement optimizations that help only one program, but hurt the performance of other programs. everyone The Go compiler used to be in a similar situation, up until Go 1.20 introduced Profile-Guided Optimization (PGO). By recording the production behavior (through profiling) and feeding that profile back to the compiler, we allow the compiler to make better trade-offs for a specific program or workload. Go 1.20 introduced Profile-Guided Optimization (PGO) profiling for a specific program or workload We think using profiles to optimize for specific workloads is a promising approach for further Go Protobuf optimizations. The Opaque API makes those possible: Program code uses accessors and does not need to be updated when the memory representation changes, so we could, for example, move rarely set fields into an overflow struct. Migration You can migrate on your own schedule, or even not at all—the (existing) Open Struct API will not be removed. But, if you’re not on the new Opaque API, you won’t benefit from its improved performance, or future optimizations that target it. We recommend you select the Opaque API for new development. Protobuf Edition 2024 (see Protobuf Editions Overview if you are not yet familiar) will make the Opaque API the default. Protobuf Editions Overview The Hybrid API Aside from the Open Struct API and Opaque API, there is also the Hybrid API, which keeps existing code working by keeping struct fields exported, but also enabling migration to the Opaque API by adding the new accessor methods. With the Hybrid API, the protobuf compiler will generate code on two API levels: the .pb.go is on the Hybrid API, whereas the _protoopaque.pb.go version is on the Opaque API and can be selected by building with the protoopaque build tag. .pb.go _protoopaque.pb.go protoopaque Rewriting Code to the Opaque API See the migration guide for detailed instructions. The high-level steps are: migration guide Enable the Hybrid API. Update existing code using the open2opaque migration tool. Switch to the Opaque API. Enable the Hybrid API. Update existing code using the open2opaque migration tool. open2opaque Switch to the Opaque API. Advice for published generated code: Use Hybrid API Small usages of protobuf can live entirely within the same repository, but usually, .proto files are shared between different projects that are owned by different teams. An obvious example is when different companies are involved: To call Google APIs (with protobuf), use the Google Cloud Client Libraries for Go from your project. Switching the Cloud Client Libraries to the Opaque API is not an option, as that would be a breaking API change, but switching to the Hybrid API is safe. .proto Google Cloud Client Libraries for Go Our advice for such packages that publish generated code (.pb.go files) is to switch to the Hybrid API please! Publish both the .pb.go and the _protoopaque.pb.go files, please. The protoopaque version allows your consumers to migrate on their own schedule. .pb.go .pb.go _protoopaque.pb.go protoopaque Enabling Lazy Decoding Lazy decoding is available (but not enabled) once you migrate to the Opaque API! 🎉 To enable: in your .proto file, annotate your message-typed fields with the [lazy = true] annotation. .proto [lazy = true] To opt out of lazy decoding (despite .proto annotations), the protolazy package documentation describes the available opt-outs, which affect either an individual Unmarshal operation or the entire program. .proto protolazy Next Steps By using the open2opaque tool in an automated fashion over the last few years, we have converted the vast majority of Google’s .proto files and Go code to the Opaque API. We continuously improved the Opaque API implementation as we moved more and more production workloads to it. .proto Therefore, we expect you should not encounter problems when trying the Opaque API. In case you do encounter any issues after all, please let us know on the Go Protobuf issue tracker. let us know on the Go Protobuf issue tracker Reference documentation for Go Protobuf can be found on protobuf.dev → Go Reference. protobuf.dev → Go Reference By Michael Stapelberg Michael Stapelberg This article is available on The Go Blog under a CC BY 4.0 DEED license. This article is available on The Go Blog under a CC BY 4.0 DEED license. The Go Blog The Go Blog