Emergence and higher order thinking

[Review other tutorials from my table of contents.]

I recently had an epiphany, so naturally my first thought is to go public and write it all out.

If you're tracking my LinkedIn narratives then you've probably read about my trip to Boston last month to attend Tech Ex 24. One of the recurring themes that came up throughout the conference had to do with emergent properties of TCP over long, fat networks (LFNs or "elephants").

I recall long hours of lecture in advanced computer networking I sat through in grad school. We spent time reading various ACM publications from Van Jacobson and others. We discussed the difficulties involved in signalling flow control within the TCP stream, how to make best use of available bandwidth. I read the texts, mastered the material, passed the tests, and over the following decade or so, I never really had much opportunity to put the knowledge to practical use.

Re-engaging the intricacies of TCP over the week of Tech Ex 24 brought me face to face with these concepts, giving me an opportunity to re-examine my perspective. The pivot for me hinges on this word, emergent. I use the word to describe second- or third-order effects that arise from underlying systems.

An emergent property of TCP's in-band flow control over LFNs is that packet loss in the order of 0.03% can destroy optimal throughput.

How's that? It has to do with Jacobson's innovation: linear growth and exponential decay. It looks like a saw tooth wave, with a gradual slope leading to maximum effective throughput, followed by a steep drop off once the ceiling is detected.

For its time, TCP was a brilliant innovation. Recall that TCP comes from a time when TDM circuits (phone trunks and their digital successors) formed the long backhaul of the early Internet. One of the properties of TDM circuits is the guarantee of bandwidth. Packet switching technologies such as Ethernet do not behave the same.

In the higher ed space (where I've spent most of my career), we innovate and collaborate and come up with some crazy ideas to overcome budget limitations. One of those crazy ideas is a transcontinental fiber network devoted to research and education: this becomes our LFN battleground. What happens to TCP when there's 100Gbps Ethernet bandwidth available, but due to geography and the limitations of the speed of light, the latency becomes a liability?

Can you imagine the frustration of the first users to discover that TCP fails so miserably under LFN conditions? What's happening and why? AARRGGHH!!

Optimize within or replace altogether?

There's an entire body of thought on how to optimize TCP over LFNs within the original TCP specification: one innovation I learned about at Tech Ex is to establish multiple parallel TCP streams to "pack" the LFN with full bandwidth utilization. I attended a workshop at Tech Ex where the presenter demonstrated various methods for invoking TCP optimizations within the Linux stack. (For example, see BBR - there were others mentioned, but I didn't take good notes ... )

On the other hand, some groups are putting time and effort into rewriting TCP, perhaps replacing it with a protocol that performs better under the conditions of contemporary LFNs. When I interviewed at Google's Mountain View HQ in 2010, I learned that Van Jacobson was employed there. Ironic? Or as a consequence? Google has put huge R&D in QUIC, a UDP-based streaming protocol.

Let me put a pin in the rabbit trail of optimizing vs replacing TCP - since my point here is to explain the epiphany ... but apparently I'm getting lost in setting up the context. (If you know me IRL, then you know this is a common struggle for me.)

Putting it all together

Hindsight is crucial here. From their original perspective, Jacobson et al came with TCP as a discrete specification that deals with a finite set of circumstances. And yet, the emergent behavior of TCP over LFNs developed anomalies that Jacobson did not foresee. (Nobody expects the Spanish Inquisition!)

Looking back from our current POV, it seems obvious how TCP breaks down over high speeds and long distances - but if it were so obvious, then I imagine Jacobson and crew would have seen it coming!

What I'm struggling to put into words here, is that second-order and third-order effects are complex and difficult to anticipate. In fact, I assert that even with hindsight, these higher order effects are sometimes difficult to detect.

I work with a retired Air Force CMSgt (E9) - talk about someone who's learned to manage the managers. This guy thinks. He coaches me all the time about second- and third-order effects - think about the downstream effects this design will have on our organization.

These two thoughts are connected. That's my epiphany. Emergent properties are 2nd and 3rd order effects that come up from an underlying cause within the system, whether or not they're foreseen by the original designers.

Conclusion

Is there a clear takeaway from this meditation? I don't know yet. I encourage you, my dear reader, to consider the consequences of your designs as you do your good work in the world. What if short term gains come at the cost of long term flourishing?

At face value, I guess I'm saying you should find a retired Air Force sergeant to life coach you through your career, and modify your Unix TCP/IP stack to support BBR. ??

What takeaways emerge for you? Please post your insights in the comments!

Happy hunting!