Simplifying Conceptual Flow Interfaces Through DDD

This is a bit of a wandering post, so please forgive me. The TL;DR is that I think dependency and coupling management at organziation boundaries strongly determines that a successful flow interface will be extremely simple—just "subscribe", "unsubscribe", flow control, and maybe some functions for security, monetization, etc.

(By the way, I am still looking for my next great opportunity to apply my 25+ years of distributed system architecture, development, deployment, and operations experience to executive sales conversations, product management, and/or platform engineering. If interested, please DM me here.)

When I wrote Flow Architectures: The Future of Streaming and Event-Driven Integration , I laid out an argument for why event streams were the future of business-to-business exchange of near real-time data, what was required to do so, and why I am convinced that this practice is inevitable. The key, I argued, was ubiquitous standards for the flow of events, including a standard for event meta-data and another for the API (or interface) to establish and maintain connections to event streams.

One of the key elements of the definition of "Flow" in this context is that it focuses on the exchange of events across organization boundaries, meaning the movement of events between companies/institutions or between divisions within a company or institution. In other words, that one organization is offering a stream of events, and another disparate organization is deciding there is value in consuming those events.

Having said that, there is an important distinction to be made here—with huge implications for architecture of your software systems. And it all boils down to a combination of concepts articulated in Domain Driven Design (DDD) and past "componentization" methodologies (such as object oriented architecture (OO), service oriented architecture (SOA), and so on).

Scale and Bounded Contexts

One of the critical concepts that DDD delivers is that of the bounded context. The term defines the logical "space" in which a common terminology and mental model apply for provided solutions. For example, in many software applications that use graphs to represent data, the term "node" might have a different connotation than it does in datacenter architecture. Identifying bounded contexts allows for better understanding of both software and human organization strategies.

Software can be designed in a way that takes the differences in language, function, and skills applied in each bounded context. For example, components that only communicate with other components and services within that context can have much tighter coupling than components (or services) that communicate with software outside of the context. Knowledge of implementation, operations, and infrastructure can be exposed more easily to other components that use the same language to describe data elements, for example, than to components that may misinterpret what those data elements mean.

This is very similar to a key rule of thumb we identified in object-oriented programming, microservices architectures, etc., namely encapsulation. By being careful about how we expose implementation details to unknown future consumers, we create greater adaptivity to changing conditions over time. We also make it much easier to optimize cost, performance, and value.

Interestingly, these concepts can be "nested" in a way that allows for encapsulation at different scales of abstraction. The DDD concept of "subdomains" is key here: within a larger domain (say a corporation or business division), there may be several subdomains (such as accounting, sales, a given product group, etc) which may, in turn, each have more that one bounded context.

By carefully managing interfaces, protocols, and dependencies at each of these scales (between domains, between subdomains, and between bounded contexts), complex distributed systems can be much easier to create, maintain, and consume.

Tricks for Managing Dependencies at Different Scales

As noted above, much of creating successful architectures in distributed systems centers around dependencies. Within a bounded context, the need to avoid coupling is less strict. In fact, it is not a problem at all for a service to directly interface with a messaging/event bus or a data store. And, of course, the closer in scale we get to the programming language function, the more liberty we have with respect to dependencies and breaking encapsulation.

However, if you create, say, an accounts payable service with an API that exposes technical details of how its data store works, you create a situation in which the consumers' developers have to learn all of context to use the service. Likewise, if you create an API in which the accounts payable service needs to understand details of any consumers' implementation in order to be able to serve them, you force each consumer to share those details before receiving service.

Most of us know this well. We are two decades into an era in which doing business on the Internet has forced most software developers to understand they have to be smart in the ways they encapsulate their services and applications. However, there are a few things that apply at every scale that are pertinent here:

1. Never directly expose data structures. While APIs can and should provide access to any relevant data, they should not do so by directly exposing variables, data stores, or other "systems of record". There should always be some software functionality, even if it is trivial, that accesses and packages that core data for the user. Most of the time the functionality will be non-trivial, providing key capabilities such as access control, security validation, or data transformation.
2. Avoid directly exposing implementation. This is similar to (1) above, but it extends to hiding what tools are being used, what algorithms are executing, and so on. If you are using a cloud service for message passing within an accounts payable service, for example, the user should not have to learn the publish-and-subscribe interface of the tool you selected. Again, wrap that with functionality that maps that interface to terms an functions that apply to the accounts payable context.
3. Do not require clients to expose implementation details. Asking someone calling your service to provide links to their internal implementation components is a pretty obvious no-no at this point, but in the past there were examples of services requiring connection to the client's message queue or data store in order for the service to complete its task.

There are some other rules that apply to good dependency management and decoupling, but these are the critical ones for the point I am making today.

Dependency Management and Flow Architectures

When providing event streams across organization boundaries (generally domain or subdomain boundaries, but these equally apply to passing events between bounded contexts), all three of those rules of thumb are critical. And, when you think about it, they have major implications for how flow interfaces and protocols should be implemented.

First, in order to avoid exposing implementation details, every event should be mapped to both a metadata and payload protocol that consumers can decode and use. This means one (or both) of two things:

• You've published a schema for your event data that consumers can use to parse the event
• You use well known standard schemas that are readily available for consumers

I believe, at least at first, most flow systems will do a combination of the two: use CloudEvents as the standard metadata protocol so consumers can understand the context and form of the event, but publish schemas for their payloads so that consumers can interpret the details.

A Flow Interface Will Be Strongly Decoupled

Interfaces should then be designed to make acquiring these events as simple as possible without exposing data or implementation directly. I believe this is a profound requirement, and one that strongly dictates the form a flow interface should take.

Namely, in order for flow interfaces to properly hide implementation details in the producer, it should assume the simplest form of event stream that covers nearly all possible use cases. When you look at the diagram below, that can only be an atomic single event, although with metadata that provides the context to map it to other forms of processing..

Why? Because anything else exposes an implementation detail that may not apply to those other forms of processing. An interface that assumes events are to be processed as a group (or series) will not be useful to those that want to process atomic events. An interface that exposes workflow capabilities will not apply to a large number of immediate action use cases.

I propose that a future flow interface should be extremely simple: the ability to subscribe to a stream, the ability to flow-control that stream (e.g. start and stop as needed to not be overwhelmed), and the ability to unsubscibe from the stream. Maybe some additional abilities for security, monitization, etc, but the core stream management functions would be extremely simple.

This is not a fully formed idea, so what are your thoughts? If we were to create a ubiquitous standard for getting access to event streams in other organizations, what would you assume had to be there?

I look forward to your comments. As always, I write to learn.