Protocol Buffers (Protobuf) explained!

Let’s explore together what Protocol Buffers are and what they can bring to your projects!

Protobuf in short

• What: Protocol Buffers is a data format used to serialize structured data.
• Who: It was developed internally by Google before being made available as free and open-source.
• When: The first version was created in 2001 (about the same time than JSON), but it didn’t reach the public before July 2008.
• Why: It was designed to be used instead of XML while being both smaller and faster.
• Where: it can be used everywhere XML or JSON can be used, but it is often associated with Remote Procedure Call (RPC) system and network communication in general.
• How: it relies on a definition file (.proto) that describe the structure of the data. We will go deeper into details later in the article.

The .proto file

What it contains

(Example based on the following file)

A message is a structure containing fields and eventually enum or nested message declarations.

Each field has :

• a cardinality indication (optional, required or repeated),
• a type (like string, uint32, a message, an enumeration, …),
• a name,
• a field number (it must be unique for a given message).

Repeated fields are always optional.

What is present in the serialized data?

Like JSON which is key-value based, each record contains both the field it is associated with and the value. The first difference being that the field isn’t identified by its name but by its number.

If a field is optional and a value wasn’t set, the corresponding record is just omitted from the binary data to reduce its size.

As of repeated fields there exists two formats:

• packed, a record contains the field id, a length indication (the number of values), and then the values (mostly used for numeric types),
• unpacked, there is a record for each value.

Some field type values like string and bytes also are preceded by a length indication. (It’s much more efficient to retrieve a string value by reading n bytes than reading char by char until the first non-escaped double quote character like it would be done for a JSON file)

Most integer values use a variable size representation

It’s similar to utf-8 string encoding where each char is encoded as 1 to 4 bytes.

In protobuf 64 bits variable sized integer values are encoded using 1 to 10 bytes.

The numeric values that does not use variable size are float, double, fixed32, sfixed32 (signed fixed 32), fixed64, (signed fixed 64).

The pros and cons of using protobuf

The pros

First the resulting binary format is quite light compared to an XML or a JSON equivalent for the same content.

Also, using the .proto file with the protoc (Protocol Buffers compiler) we can generate very efficient code in various programming languages to serialize and deserialize data.

It is strongly typed, which ensures safety.

The schema can evolve as long as previously used field numbers are respected, the compatibility if both backward and forward! (unknown fields are just ignored)

The cons

As it uses a binary format, it isn’t human readable.

Also you can’t do anything if you don’t have the schema file (or generated source code from it).

It can be too much overhead to use it for small projects or when a small amount of data is transmitted.

That’s all folks!