Why I hate APIs

APIs are not for Application Programs, they are for Application Developers

In a previous post, I put forward the case for sharing data so that we can solve collective problems cooperatively. By cooperatively, I mean human-to-human, machine-to-machine and any combination of the two, including human-in-the-loop systems and agent-based systems with AI and ML. The will to tackle collective problems, climate change, net zero and sustainability among them, have been gaining momentum - so what’s holding back the collective endeavour to solve them?

The surprising villain of the piece: the humble Application Programming Interface or API.?

Dealing with the complexity in cooperative ecosystems

First we have to address the concept of complexity, and start with an axiomatic truth.

All real-world problems are complex. You can address the problems of complexity at source (i.e. the producers’ job) or at the destination (the consumers’ job) or a mix of the two. Unlike the real-world, in a bottled world, a closed ecosystem, you can make either choice as you own both “ends” of the problem. In an open ecosystem, can you - should you - push the complexity onto the others in your cooperative enterprise, especially if the “others” are computers and their programmers?

Where is the complexity?

A nice analogy for thinking about where complexity (and the work to address it) sits in an interaction between two parties is flat-pack furniture. IKEA sells you a bookcase. IKEA could just ship you a bookcase. Or, in the opposite extreme, they could ship you a tree, a saw and a chisel. Instead, they choose a middle ground where they add as much value to the product as they can, package it up in a convenient way and, crucially, put some instructions in the box to tell you what all the bits are and how to put the bookcase together. This approach distributes the complexity and the work, hopefully reducing the total collective work done in making, packing, shipping, assembling.

Optimising for re-use and interoperability

The flat-pack analogy has another interesting angle. It allows for re-use - you can “hack” the bookcase and turn it into a cupboard by mashing it with other bits from IKEA as their screws and hinges are interchangeable. Obviously a tree is reusable as you can make anything from wood pulp to a canoe out of it, but that’s a lot of work.

Complexity in APIs

In theory, the solution to complexity already exists - complexity is hidden behind APIs and they should be designed to be trivial for consumers to consume. Unfortunately, in the harsh realities of the real world, APIs are seldom like this. They are routinely developed for a single purpose and rarely designed to be discovered and consumed by other machines. This single-purpose nature is unfortunate when you want to consume the API for something other than its original intention and good luck trying to do it in any automated way.?

What does putting the complexity in the wrong place cause?

Let me give you an example from my career. The (fortunately no longer with us) FAA airports API. This was a REST API that allowed you to query the weather and status of any airport in America via its FAA code. First of all, why FAA code, not IATA or ICAO code? The API returned time in local time (but no location so you had no idea of timezone) and the temperature was in Fahrenheit. Why does *my* application have to contain all this extra programming? Look up tables, backward location cross-referencing and time and unit conversions. This API was definitely not designed to be discoverable by another program and had no thought to reuse. It left so much for the consuming application to do and forced me to code the logic necessary to deal with the complexity. This was giving me a “tree and a saw”.

Why Hate APIs? They’re how the IT industry runs

My heart goes out to Roy Fielding. In 2000, in his seminal PhD thesis he set out the backbones of what has now been accepted as the de facto REST API “standard”. This paragraph shows his ambition to create a “uniform interface”.?

“In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state”

The highlighting is mine, Fielding was setting out a way to allow the API to be understood by the consumer, especially another computer.

Towards an answer: HATEOAS and why it didn’t work out

Those words have been condensed into the horrible acronym of HATEOAS (“hate-ee-ohs” is how I pronounce it) and it’s very divisive in the developer community. HATEOAS has always seemed to me (at least) to be a good idea. At its core, it’s a way of making APIs “discoverable” where by “discoverable” we mean that digital agents (computer programs) can start at the root of the API and discover all the branches in a unified way. For example, in a ridiculously over-simplified banking application, GET https://bank.com and the returned HTML will contain “rels” (relative links) which will point the application to the correct URL to add, delete, update and account. The application can crawl the rels and figure out how the API is put together - all by itself.

Fielding as also been quoted saying:

“Unfortunately, people are fairly good at short-term design, and usually awful at long-term design”

So, guess what? This short-termism has been why the idea of HATEOAS hasn’t ever taken off and we’ve collectively ended up in the current “API Mess”. Putting the “rels” in the code and writing apps that could cope with that was too much trouble in the short term, so people didn’t bother or didn’t understand and whole sectors have grown up around clearing up the mess that shouldn’t have been there in the first place.

Isn’t Open API the solution?

Is REST the “uniform interface” that Fielding desired in his PhD Thesis? On one level, yes it is. It’s uniform in that you use the Internet standard of HTTP to access the API, but that’s as far as it goes. The structure of the URLs, the return type of payload data and the behaviour of the verbs: GET/PUT/POST/DELETE/HEAD, etc are largely left to the designer of the API. “What about the Open API specification?” (I hear you cry). What about it? It will generate documentation and code stubs, but they are for the application *developer* at development time, not for the application *code* at run time.

Go on then, find an interoperable API. I challenge you.

The challenges around APIs are compounded for open world use-cases. How do you find them in the first place? That’s before you can even try to understand what they’re about and what they deliver. There are, of course, methods to address this, for example, Open Data aggregator sites like https://data.gov.uk/. This site is hard to use even as a human as it mixes formats, data and APIs without much control over what you get. But of how much use, truly, is a PDF file of farming results in 2014? https://www.programmableweb.com/ is better, but if you have to write a crawler to find the API and then write a crawler to understand the API we’re not exactly hiding the complexity, are we? In the closed world, you could have a database of API endpoints, but... in a closed world.

What about GraphQL and GRPC?

These are new technologies for APIs and seek to replace REST. They do have some good points: GPRC is good for streaming and GraphQL does consider metadata, but they still suffer from the same problems of findability and discoverability. They are different mechanisms for APIs that don’t solve the underlying problem: APIs don’t solve open-world problems.

Is there an answer?

I try not to be a grey-beard veteran who just rants about “The Industry” and how things were much better when you had to write things in System 360 assembly language, so I’m going to try to give you an answer. It might not be complete, but this is fairly uncharted territory and that, by definition, means you don’t have a map to follow.

Let’s have a look at the other things Fielding wanted?

• Identification of resources?
• Manipulation of resources through representations
• Self-descriptive messages

No surprises what I think is the answer (and I’m not the only one. The Fraunhofer Institute and Ericsson agree on this video). I think semantically-described digital twins operating in ecosystems is the way forward. Especially if you think of digital twins as “nano-API” for one instance of a resource type. A ship, a sewage tank, a person, whatever, but only one of them, not a whole bunch.

Let’s address the points from above one-by-one

Identification of resources: Let’s use the W3C standard for Decentralised ID to generate a self-sovereign ID and then use this as the “subject” for subject-predicate-object triples in RDF. That gives us identity and description.

Manipulation of resources through representation: If you make a digital twin of each resource - that’s your “representation”. The manipulation bit comes from adding a control interface to the twin so you can send it messages to change its state (and therefore the state of the underlying real-world object - if there is one)

Self-descriptive messages: More semantic metadata. This time, for the contents of the messages come from the twin. If you’re worried about message bloat with all that metadata, separate the metadata and the data and send the metadata of the feed once on subscription and then send the messages that match the metadata.?

Where does that leave us? At IOTICS, we have an axiom: “Metadata is more important than the API”. With the richness in the metadata, your API simplifies. This simplification is the essence of the “data-centric” approach. Your API becomes

• Search - find the twins that are interesting to you (or your program)
• Describe - get them to tell you (or your program) about themselves and their capabilities
• Bind - ask to follow their feeds or attach to their controls (this is where authentication and authorisation kick in)

Over millennia, we humans have evolved human-to-human interfaces via body posture, gesture and language. Our communication, therefore, is ambiguous, time-consuming and fraught with error. Let’s try not to repeat the mistakes when we include our digital allies in the conversation.