Protocol Buffers with Golang

In this article, I will cover Protocol Buffer briefly and how can we use it in Go Programming language. I am assuming that you are already familiar with basic Go programming.

What is Protocol Buffers?

Protocol Buffers (also called Protobuf) is a method of serializing structured data. It is language-neutral, platform-neutral, extensible mechanism for serializing structured data. It is like XML or JSON, but smaller, faster, and simpler. You define how you want your data to be structured once like a template, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. It is useful in developing programs to communicate with each other over a wire or for storing data.

Why should we use it?

Protobuf is -

• Smaller: It is 3-10 times smaller than XML or JSON. If you are storing data then it will reduce storage space. If you are using it as RPC then it will reduce latency as message size will be much smaller.
• Faster: Serialization and deserializations is 20-100 times faster which will increase performance of your system as it is using less time in serialization and deserializations (source).
• Simpler yet extensible: It is simple to implement and start with. It is very easy to update or extend an existing protobuf message file.
• Language and platform-neutral: Protobuf messages are language and platform independent. That gives you freedom to choose technology of your choice for different entity using this protocol. For example your client can be using C# but it can still communicate with your server written in Java through protobuf.

Protobuf with Golang -

Both Protobuf and Golang are from Google and Golang has official support of Protobuf. There are 3 simple steps to use Protobuf in any language -

1. Define message formats in a .proto file.
2. Use the protocol buffer compiler (protoc).
3. Use the Go protocol buffer API to write and read messages. 

The protocol buffer compiler generates source code that contains language specific data structure for defined messages of .proto file.

You can install protocol buffer compiler (called protoc) from here - https://github.com/google/protobuf

Let's take an example of implementing an application that encode and decode message using protobuf-

First step would be definition of message.

message User {
  string name = 1;
  int32 id = 2;  // Unique ID number for this User.
  string email = 3;
}

We will save it as user.proto. This is a contract between all entity that wants to use this User message. Now client and server can use their language specific protocol buffer compiler to generate source code. For Golang plugin for protoc, run this command -

go get -u github.com/golang/protobuf/protoc-gen-go

For second step, we will use this command -

protoc -I=$SRC_DIR --go_out=$DST_DIR $SRC_DIR/user.proto

Because you want Go classes, you use the --go_out option – similar options are provided for other supported languages. This will generate file user.pb.go.

For third step you need to import this generated code in you application. Here is an example of how you might create an instance of User:

import (
"include_generated_file_package"
"log"?
"github.com/golang/protobuf/proto" // for protocol buffer 
"github.com/golang/protobuf/jsonpb" // If you want to use proto to json?
"io/ioutil" // for I/O?
?)

// Create user instance
user := &User{
        Id:    123,
        Name:  "Nishant Kumar",
        Email: "[email protected]",
}

// Writing file
out, err := proto.Marshal(user)
if err != nil {
        log.Fatalln("Failed to encode user data:", err)
}
if err := ioutil.WriteFile(fname, out, 0644); err != nil {
        log.Fatalln("Failed to write user data:", err)
}
?
// reading file
in, err := ioutil.ReadFile(fname)
if err != nil {
        log.Fatalln("Error reading file:", err)
}
user := &User{}
if err := proto.Unmarshal(in, user); err != nil {
        log.Fatalln("Failed to parse user:", err)

If you want to convert protobuf message to JSON then don't use default JSON encoder of Golang. It will not be able to convert protobuf message to JSON correctly. Instead use jsonpb that is part of protobuf library. Similarly use official protobuf library instead of some third party.

user := &User{}

// JSON to Proto
?if err := jsonpb.Unmarshal(in,user); err != nil {
		log.Fatalln("Error converting JSON to proto:", err)
	}
// Proto to JSON
ma := jsonpb.Marshaler{}?
err = ma.Marshal(out, user)

Marshaler has some option to customize output.

I hope that you will find it useful.