Design as Code: A Frictionless Low-Level Design Pipeline
A client of mine is building a web application that features a complex data model, supported by a collection of APIs. My client has asked me to review their design & development processes and help them achieve a better outcome for the project.
As part of the discovery process, I asked the tech lead to show me a design artifact describing one of their APIs.
An Example By Negation
For the uninitiated: an Application Programming Interface (API) is a set of rules and protocols that allow one component of a software application to interact with other internal components or with another application. In the hierarchy of abstractions leading from an application’s business requirements to its implementation in code, an API design sits right in the middle:
Back to my story…
It took a couple of minutes to clarify my request with my client’s tech lead. But after some internal discussion, one of his developers shared his screen and brought up the OpenAPI specification for the API in question.
If you’ve never seen an OpenAPI (a.k.a. Swagger) spec, here’s an example:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Swagger Petstore",
"description": "A sample API that uses a petstore as an example to demonstrate features in the swagger-2.0 specification",
"termsOfService": "https://swagger.io/terms/",
"contact": {
"name": "Swagger API Team"
},
"license": {
"name": "MIT"
}
},
"host": "petstore.swagger.io",
"basePath": "/api",
"schemes": [
"http"
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {
"/pets": {
"get": {
"description": "Returns all pets from the system that the user has access to",
"operationId": "findPets",
"produces": [
"application/json",
"application/xml",
"text/xml",
"text/html"
],
...
},
...
},
...
},
...
}
… only theirs went on vertically for about a thousand lines.
A thousand-line JSON Swagger specification is not an API design artifact!
In this article, we will discuss:
Design Artifacts Redux
In a previous article I went into some detail regarding the theory and application of artifacts in a “mechanical” design process.
I described a complete design artifact as a matched pair of work items:
Beyond providing a useful balance of clarity and detail, this double presentation constitutes a source of internal consistency that can be used to validate the design artifact against error.
I also described an external consistency between design artifacts at different levels of abstraction:
Taken together, these two forms of design consistency lend themselves to an iterative process that should be familiar to every Agile practitioner: grooming the design backlog.
An Aside: The Design Backlog
Backlog Grooming is an essential part of the Agile cycle.
It’s also a lot of work! Most projects schedule backlog grooming weekly at least. It’s a 1-2 hour meeting where participants pull a pile of issues out of the project backlog and…
With all that already in play, the last thing any Scrum Master wants is to find he has to groom another backlog.
Not the idea! Everybody take a breath.
In a healthy Agile project, there is only ever one backlog!
This only makes sense. You have one team, focused on producing one body of work. Your backlog is simply the pile of stuff that hasn’t been done yet. As we saw above, an essential part of the backlog grooming job is breaking the complex bits of stuff down into simpler stuff, and a single backlog makes it much easer to maintain a trace on which bits came from which other bits. Life is hard enough.
And yet… there are different kinds of stuff, aren’t there?
In A Modern Agile Project Manifesto I proposed a breakdown of the various issue types available to a project. I did this within the context of a Jira implementation, but that’s really beside the point.
The lessons here are:
That last point is the important one. We’ve beaten the design horse half to death in recent weeks, and if you’ve noticed any consistent theme it should be this one:
Design is not the same thing as implementation.
Humans are bad at multi-tasking, so stop. If you have a two-hour backlog grooming meeting every week, simply splitting it up into two meetings—one focused on design and the other on implementation—will make both meetings more focused, more productive, and much more effective in terms of project outcome!
Stories & Bugs are both arguably about implementation. What about Epics & Tasks?
Epics are admittedly weird. Let’s revisit the definition of an Epic from A Modern Agile Project Manifesto:
An Epic represents a very high-level grouping of other issues. Other issues should come out of the Backlog and be resolved in a single Sprint. If they can’t be, then they should be broken down until they can. Epics endure across several Sprints, and generally represent a project phase with respect to a broad area of functionality. Epics can be organized into a Roadmap expressing the Project’s long-term plan. Most Epics are never resolved, because the functionality they represent is always subject to future development. If not, an Epic is resolved when all the issues it contains are either resolved or moved to another Epic.
This suggests that an Epic can be used as a container of both design-related issues (Questions) and implementation-related issues (Stories & Bugs). This would place the Epic firmly on the design end of the project hierarchy, and so Epics should be groomed in the same meeting as other design-related issues.
If you define Epics differently, you might come to a different conclusion. The important thing is to have the definition that leads to the conclusion.
Tasks fall into a different category. They tend to be procedural in nature, more focused on project management than product development. I like to isolate task grooming from design and implementation, both to keep the focus tight and to make sure the actual business of the project gets the attention it deserves.
A 30-min weekly grooming of the task list is usually sufficient for my projects.
One more very important point:
Backlog grooming is not sprint planning!
When grooming your backlog, focus is a virtue. When planning a sprint, you need to be able to see the whole backlog, so work can be assigned across the different issue types! If you ever wondered just why those two tasks are handled in separate meetings, now you know.
A Thousand Lines
By now it should be crystal clear just why that thousand-line Swagger specification I described above is not an API design artifact: because, even under the most generous assumptions, it’s not quite half of one!
Within the framework we’ve developed here, a complete design artifact has two parts:
Clearly the Swagger specification does not include a drawing.
More generally, as it represents a low-level technical abstraction (an API), we would expect this design artifact to articulate its relationships to:
It is fair to observe that none of these details actually belong in a Swagger specification! A Swagger specification is a contract between the API and its consumers, and as such it should be focused on the interface of the API, not its implementation nor its relationship with those consumers.
Being a design artifact is not what a Swagger specification is actually for.
You might argue that a Swagger spec should be included as part of a design artifact’s document, but consider these points:
This is why Swagger specs are usually generated from code comments or other machine-readable sources. They are not intended to written by humans, and they are certainly not intended to be read by humans.
You might also argue that a simple HTML plugin can turn a complex Swagger specification document into a very pretty documentation UI like this one:
That’s as good as a design drawing, right? No, it isn’t.
As a piece of documentation, this widget offers a huge improvement over the raw Swagger spec! But the information it presents is still strictly limited to the interface contract laid out in the underlying JSON document. It tells us nothing about how this API relates to other technical abstractions in the project, nor to the project’s implementation in code.
The Swagger UI is no more a design drawing than is the Swagger spec itself.
I do in fact include the Swagger UI in every API design artifact! But not as a drawing. It’s part of the document associated with the design artifact, and I include it because it is an attractive and efficient way to document the relationship between the design artifact and its implementation.
So there’s the big reveal regarding the thousand-line Swagger specification presented as a design artifact by my client’s tech team: it was almost certainly generated with some kind of automation from code that had already been implemented, and was no more an deliberate artifact of design than I am my own grandfather.
领英推荐
For many teams, the first step toward improving their design process is admitting they don’t have one.
Low-Level Design For Real
In another article we explored an example of high-level design from the VeteranCrowd project. We immediately had to deal with the following reality:
Really useful design artifacts are specific to the problem they’re solving, and proprietary to the organization that’s solving it!
In that case, it meant we had to do some pretty heavy redaction in order to protect the client’s interests.
In this case, we’re a bit more fortunate: because VeteranCrowd’s microservice architecture is extremely distributed, some of its services are considerably more generic than others. For our example—and with VeteranCrowd’s blessing—we will choose the api-message service, which their application uses to compose and send templated messages to users and is about as generic as it gets.
No secrets here!
The api-message service is implemented as an AWS CloudFormation stack using the Serverless Framework (a design pattern addressed in detail in higher-level design artifacts) Despite its hyper-generic nature, the service offers the following interesting features, all of which are detailed in the design artifact below, both in the drawing and its associated document:
The api-message service handles message-generating events raised by most of the other 15 services currently running in the VeteranCrowd back end, and it calls the aws-template service (not detailed here) in order to generate message content based on these events.
Look for comments like these on the design artifact below to learn more about specific features! Also look for links to PlantUML source code in diagram captions. We’ll discuss PlantUML in a later section!
I’ll also call out key principles like this.
One last point before we dive into the design artifact: what I am showing you below is a preserved snapshot of a live artifact!
Since the project’s requirements and implementation are in a constant state of flux, all of our design artifacts (including this one!) will have points where they are incomplete, internally inconsistent, or out of sync with other design documents and the implementation. This is par for the course!
The purpose of the design process is create a structured framework where we can…
That’s the job! Let’s get to it.
api-message Design Artifact
LinkedIn articles are a little format-limited, so I won't be able to reproduce this design artifact here. Instead, click here to see it on my blog!
Design As Code
A basic principle of economics is that the more expensive anything is, the less of it you get. There’s no such thing in life as a free lunch.
As in life, so in design!
In Turning the Crank, I showed you a few very fancy high-level design diagrams like this one:
I created this diagram and the other high-level artifacts in that article with LucidChart. LucidChart is a great tool for creating all sorts of production-quality diagrams, and I highly recommend that you use it as seldom as you can possibly get away with!
High Touch: High Cost
As we observed in that article, high-level design is generally a business-facing activity. The entities described in high-level design artifacts are business entities, and the language used is a business language. Most critically, the audience for high-level design artifacts is generally a business audience!
A business audience is a different breed from a technical audience. They care about substance… but they tend to be far more sensitive to presentation and production qualities than the average bunch of coders. Business audiences want to see pretty pictures that tell a story.
The audience for high-level design artifacts is a business audience.
Moreover, we observed in that article that in a healthy project, the design artifacts are the project documentation.
At a low level, this means that great design artifacts describing technical abstractions make great technical documentation for a technical audience. At a high level, your business-facing design artifacts are likely to find their way in front of senior leadership, business partners, and prospective investors.
For that kind of audience, a certain degree of polish is worth the effort. And the good news is that your high-level business processes and relationships will change much more slowly than your low-level technical abstractions… so even though creating and editing fancy diagrams with tools like LucidChart is a pain in the behind, you won’t need to do it that often.
Gearing for Speed
Low level design is very different.
For one thing, there’s a lot more of it. In a typical project, you might have a dozen or so high-level design artifacts, but hundreds of low-level design artifacts… none of which are of any real use to you if they fall very far out of sync.
Also, the farther you descend the ladder of abstraction, the more often things change. Nothing influences a design quite like another pair of eyeballs, and once you commit to visual design artifacts and regular design reviews, you’re going to get a lot of eyeballs on your work. Expect your design repository to evolve, and fast!
Nothing influences a design quite like another pair of eyeballs.
But here’s the good news: unlike high-level design artifacts, the audience for your low-level design artifacts is a technical one. They will care far more about the substance of your design than they will about its production qualities.
In fact, just the reverse is true. It’s neat to see a design laid out in a diagram one can grasp at a glance… but to see that diagram change regularly in response to feedback is downright impressive! It means your team can reliably catch mistakes in design review before they hit the codebase, and before anybody has to burn the midnight oil coding the wrong components.
So your design process is in place, and your technical audience is chomping at the bit to iterate on your design artifacts… and the entire program will come crashing to a halt if you can’t pump new design drawings out as fast as your team can review them.
In low-level design, pretty is optional, but speed is essential!
So let’s talk about your design pipeline.
The Design Pipeline
Scratch a typical software project’s design repository, and for every diagram you find you’ll see a dozen spreadsheets and a hundred text-based documents.
And why not? Economics strikes again: typing is easy and cheap, so you get a lot more of it.
If you want your team to produce as many design drawings as text documents, you need to make drawing as easy as typing.
What would that look like? An ideal pipeline for text-based drawings might look something like this:
Sounds reasonable, right? That’s PlantUML in a nutshell.
Here’s an example. Remember this diagram?
Here’s the PlantUML source code that generated it:
@startuml design-external-consistency
allowmixing
skinparam componentStyle rectangle
package "High-Level Artifact" as artifactHigh #LightCyan {
component "Drawing" as drawingHigh
component "Document" as documentHigh
drawingHigh <--> documentHigh
}
note bottom of documentHigh
All issues raised
are addressed in some
lower-level drawing.
end note
package "Low-Level Artifacts" as artifactLow #Cyan {
component "Drawing" as drawingLow
component "Document" as documentLow
drawingLow <--> documentLow
}
note top of drawingLow
Every element addresses
an issue raised in the
higher-level document.
end note
documentHigh <-> drawingLow: "**//external consistency//**"
@enduml
If you use Visual Studio Code and GitHub, setting up a PlantUML pipeline is very straightforward:
At the end of the day, the goal is to frame each of your design artifacts once. After the first render, the only changes you should ever have to make are substantive changes to design drawings and document content. The rendering pipeline should handle the rest.
What you have now is Design as Code: a frictionless design process that is fully integrated with your development cycle and places design artifacts at a peer level with the rest of your codebase.
Conclusion
Design isn’t just a template we pull off a shelf. It’s the guarantee that the product you put into production is the one your business actually needs.
Design is a process that creates and maintains a chain of continuity down a ladder of abstraction that connects the needs of the business, expressed in business language and concepts, to the implementation of those needs in code.
Some of that code runs web servers, and some of it draws pictures. Choose your own adventure.
Design is expressed in a set of complete and consistent design artifacts, and those artifacts have value. But it is the process that creates those artifacts, and the disciplined team culture that runs the process, that are the true technical assets of the business.
In the end, every project team’s goal is to deliver a great product that can serve today’s business needs with room to spare for whatever tomorrow brings. That means doing some hard things, and design is one of them… unless it isn’t.
After all, isn’t that what engineers do? We make hard things like launching rockets and drawing design diagrams as easy as pushing buttons or… typing.
Do good work! ??
Visit my website for more great content & tools, all built for you with ?? on Bali!