The Perfect Protocol Packet

The last three projects I've worked on have required the development of custom data communications protocols. Whether these custom protocols run on top of TCP, UDP, BLE, or bare serial line, they aim to solve a common problem: reliably and efficiently transferring data between devices. This article defines the lowest-layer, most-basic packet format upon which we can define other protocols and features as the need arises, while maintaining backward compatibility.

We firmware and software engineers often lose traction on projects where we need to design a custom protocol. However, we needn't be slowed thinking we must define the entire protocol architecture at the beginning of our project. With a hierarchical approach, we can define a simple, extensible, ages-old basic packet-structure that will last the lifetime of our project, yet enable us to begin sending and receiving data early in our project as we develop higher layer protocols:

1. Design a basic packet that is so simple, so devoid of anything but necessary fields, that this packet will never need changing (versioning), yet it supports virtually infinite specialization and extension.
2. On top of this packet we can define many other specialized protocols. Think of this packet as a base class that other protocol classes inherit from and specialize upon. Put another way, this packet is like the Ethernet frame; simple and single-purpose; any specialization is accomplished with higher layer protocols such as IP, UDP and TCP. (In our case, the higher layer protocols will be our own design, left to later articles.)

This allows us to get data flowing early in our project, validating our physical and link-layer communications media and other underlying infrastructure (e.g., UDP over IP and Ethernet, UART over serial line, etc), and more importantly we can begin developing and testing the applications that use that data. We can add control features such as acknowledgements (positive and negative), flow-control, destination fields, chaining, versioning, etc. in higher layer protocols as we encounter the need. (Remember that these features are not part of this basic packet, but rather are part of higher layer protocols that are within the payload of this packet.)

Here is our perfect packet:

To begin sending and receiving data, we can define an initial "TEST" protocol, say, using value '0x00' for the Protocol field. Packets of type TEST can then be routed to a test application that knows how to parse that payload.

Using this packet gets us moving forward on our project early and efficiently; we can use this format as the base packet for the lifetime of the project. Next we'll need to define the format for the TEST protocol, which will reside in the payload of our base packet. The definitions of that and other protocols are for future articles, but keep in mind that we have room to experiment because we have 65,535 protocols remaining (the Protocol field is 16-bits).