Protocol Buffers: Understanding Concepts

Protocol Buffers (often called Protobufs) was developed by Google to tackle inefficiencies in data serialization. Traditional methods, like XML and JSON, were slow and bulky, which affected performance and data handling. Protobuf addresses these issues by using a compact binary format that speeds up serialization and reduces data size.

Note: I encourage you to read this insightful newsletter by Neo Kim , which explains how LinkedIn was able to reduce their latency by 60% simply by replacing JSON with Protobufs.

Additionally, Protobuf is language-platform agnostic and it supports multiple programming languages, making it easier to integrate and communicate across diverse systems.

Protobuf is defined in a?.proto file which contains major two components:

1. Messages
2. Services.

1. Messages in Protobufs

Messages defines the type and structure of the data that needs to be exchanged. A simple example of Message in .proto file can be looked as following:

message User {
  int32 id = 1;
  string email = 2;
  bool is_active = 3;
}        

We can see the values 1, 2, 3 which are assigned to id, email and is_active property. These numbers are known as field numbers which should be unique. They are used to efficiently encode and decode data.

When a message is serialized, each field is identified by its unique number rather than it's name. For instance, in a User message with fields like id (1), email (2) and is_active (3), the serialization process encodes these fields as binary data with their respective field numbers.

How Serialization Happens

Suppose user message contains: {id: 123, email: "[email protected] ", is_active: true}, the serialization process would look like this:

• The field number is encoded alongside the field's value. For example, for id with field number 1 and value 123, the binary representation might include 0001 as the tag followed by 123 as the value in binary.
• Each field is represented by a tag that combines the field number and the wire type in Protobuf's binary encoding. For example, for email with field number 2 and value "[email protected] ", the tag 0010 identifies the field number and type followed by the encoded string value.

Note: The wire type specifies how field values are encoded in binary data, guiding Protobuf on how to interpret the bytes that follow the field’s tag. For example, wire type 0 (varint) is used for encoding integers and booleans, while wire type 2 (length-delimited) is used for strings and other length-prefixed data.

The binary output after serialization would be like:

Tag 1: 0001 0111 1011  (field number 1, value 123)
Tag 2: 0010 0110 0110 0110 0110 0110 0110 0110  (field number 2, value "[email protected]")
Tag 3: 0011 01  (field number 3, value true)        

Nested Message

You can define Protobuf messages within other messages and use types like enums — effectively creating nested types. Here’s an example:

syntax = "proto3";

message User {
     int32 id = 1;
     string email = 2;
     bool is_active = 3;

     enum SocialMediaType {
         FACEBOOK = 0;
         TWITTER = 1;
         LINKEDIN = 2;
         INSTAGRAM = 3;
     }

     message SocialMediaProfile {
         string username = 1;
         SocialMediaType type = 2;
     }

     repeated SocialMediaProfile social_media_profiles = 4;
}        

In this example, the User message includes a nested SocialMediaProfile message. The SocialMediaProfile message has two fields: username and type, which are used to represent a user's social media account details. The type field uses an enum called SocialMediaType to categorize different social media platforms.

The User message also contains a repeated field of SocialMediaProfile messages named social_media_profiles. This means that a single User can have multiple social media profile entries.

• Protocol Buffers support enums which create a type with a predefined list of values.
• Messages can be created inside Messages.
• repeated field label is used to define repeated values (like an array :) ). On proto2, there was required and optional keywords which are deprecated in proto3.

Field Number Scope

• Field numbers are unique within the context of their own message but not across different messages.
• In Protobuf, you can use the same field number in different messages because each message handles its own field numbers independently.
• For instance, in the above example, id = 1 in the User message and username = 1 in the SocialMediaProfile message are valid and do not conflict.

Field Number Uniqueness

• While it is fine to have the same field number in different messages, it is important to maintain field number uniqueness within a single message.
• This means that in the User message, no two fields should share the same number
• Similarly, in the SocialMediaProfile message, field numbers should be unique within that message.

2. Services in Protobufs

In Protobufs, services define a set of RPC (remote procedure call) methods. These methods are like functions or procedures that you can call over a network. Services help different systems or components communicate by allowing one system (the client) to call methods on another system (the server).

Messages define the structure of the data, while services specify the APIs for accessing and manipulating that data through remote procedure calls (RPCs).

Here's a basic example of defining a service in Protobuf:

syntax = "proto3";

service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
}

message GetUserRequest {
  int32 user_id = 1; 
}

message GetUserResponse {
  User user = 1; 
}

message User {
 // above
}        

Compiling Protobufs

The .proto files can be compiled into multiple languages using the Protocol Buffer compiler, protoc. For example, to generate Python code, you would use the following command:

protoc --python_out=. user.proto        

This will generate a python file named user_pb2.py, which includes the necessary code for creating, manipulating, and serializing the defined messages.

Implementing Protobufs in Code

Here’s how you can use the generated Python code to create a User message, populate its fields, and then serialize the message into a string:

import user_pb2  # This is the generated file for the User message

# Create a User message
user = user_pb2.User()

# Set the fields
user.id = 1234
user.email = "[email protected]"
user.is_active = True

# Add a social media profile
profile = user.social_media_profiles.add()  # Add a new SocialMediaProfile
profile.username = "np123"
profile.type = user_pb2.User.FACEBOOK

# Serialize the message to a binary string
serialized_user = user.SerializeToString()        

Similarly, parsing of serialized data can be done as follows:

import user_pb2  # This is the generated file for the User message

# Assume `serialized_user` is the binary data obtained from serialization
user = user_pb2.User()
user.ParseFromString(serialized_user)

# Access the fields of the deserialized User message
print(user.email)        

Summary

Protocol Buffers (Protobuf) helps make data handling faster and more efficient compared to older methods like XML and JSON. It uses a compact binary format, which makes data smaller and quicker to work with. Protobuf is great for defining data structures and creating services that allow different systems to communicate with each other. It supports many programming languages, making it versatile and easy to integrate into various projects. That all's for this article.

Stay tuned for more insights on Protobuf and gRPC topics.

For more details and resources, visit my personal website .