Uncovering API Implementation

APIs have been around for a while across all the industries. For most organisations, APIs have become synonymous to RESTful APIs. However, APIs existed in non-RESTful fashion for many decades now. It was .h file in C and C++, interface definition in Java. These were shared with developers to consume the implementation that sits behind these interfaces without having to worry about how these interfaces are implemented. At the same time, it provided ability for implementation teams to upgrade it as long as interface didn't change. Now, when we apply this to RESTful APIs, should it change? What does implementing an API really mean?

Many organisations that I worked with in the past had decided to implement APIs as microservices using language of their choice. While this seems like an obvious pattern for implementation considering nobody these days want to create monoliths. Microservices using Domain Driven Design (DDD) provide an absolute brilliant way of implementing APIs. But let's look at the technology landscape that most of the large organisations have vs. the examples typically provided in DDD approach. A microservice is expected to be completely independent and is able manage its own data. For example, orders, billing, payments, shipping. These examples sound familiar right? However, most large organisations have a huge legacy of existing systems with a mix of open systems, mainframe, package solutions, SaaS etc. The organisations now want to figure out how to follow these "example" microservices like patterns using the legacy that doesn't quite fit into this model. This is where most of them introduce layers. A layering model similar to Service Component Architecture is one of the most popular choices since APIs evolved from Service Oriented Architecture for many organisations. Let's look at typical layers created by many organisations. These layers are known by many names depending on the creativity of the architect designing them :-). Thus, I would use just numbering to define them.

First layer

With this layering approach, the first thing organisations try to do is start exposing existing systems as REST APIs that can then be aggregated by another layer to expose more meaningful APIs for channel consumption. However, while exposing these systems as APIs, in the true spirit of adhering to microservices architecture, a layer of microservice is added on top of existing system that manages connectivity to underlying systems and often exposes the underlying system data using a canonical data model allowing APIs to be more "generic". However, many of these existing system already provide an HTTP based interface using either webservices or pure XML or in some cases even a REST API. So when the organisations implement this first layer, what are they implementing? A proxy to an existing capability just to comply with the microservices architecture? A microservice that does a simple translation from one data format to another? But, don't get me wrong. I am not trying to suggest that this layer is unnecessary. There are many cases where this layer truly adds a great value. For example, abstracting authentication and authorisation requirements for a specific system such as SaaS, connecting to a very old system that requires connection pooling and throttling to manage load etc. However, with large organisation, making a call whether this layer is required or not is often difficult and not done correctly (may be because of delivery pressure or lack of capability to take a decision at scale) resulting into many unnecessary APIs that really are point to point connections and provide more overhead than value to the organisation.

Second layer

As mentioned earlier, the second layer is the aggregation or business logic layer. This is where bulk of the problems start. In a true spirit of creating reusable APIs, many organisations decide that this second layer should implement all the orchestration and aggregation required to provide a "meaningful" API to the consumers. For example, if you want to get all the bank accounts that a customer has with corresponding balances, this layer will need to first find out list of accounts for a given customer identifier from a system (using first layer). Then for each account number retrieved, derive the system that holds the balance for the account. Then invoke API on that system (using first layer) to get the balance. Finally, aggregate the response from all the systems to create one final response to be sent back to the API consumer. Important thing to consider here is that we only discussed about "happy path" here. There are number of failure scenario that need to be dealt with. This second layer is responsible for all of this logic so that a single API can be provided to the consumers. Question that comes to my mind is, why are organisations implementing this as a REST API (ultimately as a microservice)? Isn't this a capability that needs to be implemented using best available technology regardless of whether it is exposed via REST API or some other mechanism (Events, queues, websockets, user interface etc.). Once the capability is built, API implementation should then work out a design to expose this capability using appropriate API techniques. For example, in scenario above, organisations should think about a design that keeps balance with the account numbers in the system that holds relationship between customer and accounts. That way, one system can provider complete information required for this API. Are organisations implementing System of Record (SoR) in an API? Or, are they trying to solve a wrong problem using API? The consequences of this approach are multi-fold. Some of them are listed below based on what I have seen with multiple organisations;

• The API implementation becomes too complex. API implementation is now really a mixture of business function and interface implementation. This implementation needs to handle failure scenarios both from a functional as well as technical perspective which can lead to very complex implementation.
• The API teams become "know everything" teams because they need to understand every aspect of business process (some of it may be hidden in first layer APIs), complex scenarios, failure scenarios, business operations aspects in case of failure. Imagine a "Make a payment" API that calls two first layer APIs. Debit money and credit money. Because REST APIs are typically stateless, if debit is successful and credit fails, it's now responsibility of the API to reverse the debited money and if reversal fails, handle the reversal operationally. This entire functionality is crucial from payments perspective but doesn't seem like something that "APIs" should handle.
• When you have legacy systems, not everything can be solved using microservices architecture. There are technical, organisational and security boundaries that one needs to manage. In such cases, because organisations try to implement APIs as microservices, they will try to retrofit this non-microservice friendly functionality into the microservice resulting into either an extremely complex, non-performant microservice or something that looks like a microservice but isn't really a microservice (too many functions in a single microservice)
• Because now complex business logic resides in the API implementation, it makes it error prone. In such cases, API would be termed unstable and soon everyone starts blaming APIs for their failures. Even if APIs throw a meaningful error message that may be a result of downstream system not being available, the consuming channel would still term it as the API failure resulting in loss of confidence in APIs within the organisation.
• Because APIs now do much more than exposing a business capability, it would require significantly more effort to work out requirements, design, code and test. This would increase overall cost of building APIs. This cost in reality is a sum of API implementation cost and SoR implementation cost. Thus, within the organisation you would see inconsistent cost of APIs depending on how simple or complex the business logic is.

These are some of the challenges that arise because organisation mix SoR level business logic and API implementation. This can easily be avoided by looking at API as the access mechanism for a business capability rather than business capability itself.

Third layer

Now that "meaningful" APIs are provided by the second layer, one would expect that channel applications can directly consume them to build business applications for staff and customers. However, there are two challenges still that need to be addressed;

• Where do channels keep channel specific aggregation logic? As an example, web and mobile channels need to augment order information coming from second layer API with maps data to identify location of the order. This capability is not required for orders API and the organisations don't want to duplicate the functionality in multiple channel applications. Where does this logic sit then?
• A channel application doesn't need all the data being provided by the API. Take an example of customer information API. The API providers entire customer information including all the relationships, contact history, orders etc. While this information is a must for staff facing channels, organisations may want to cut down on the information on customer facing channels. Since there are multiple customer facing channels and organisations want to provide consistent data on all the customer facing channels, where does data reduction happen?

This is where the third layer comes into picture that addresses the two issues above. However, it takes strong governance process and discipline to ensure this layer doesn't do anything other than what is outlined above. Failing to do so will result in logic spread across all over the layers and will make it difficult to debug, maintain and upgrade.

With some organisations, this layer also acts as a reverse proxy to expose the API outside the organisation where it primarily handles security aspects to ensure APIs within the organisational boundaries are well protected from external threats. However, some organisations choose to create this reverse proxy as a separate layer creating fourth layer of API.

Putting the layers together

Putting it all together, a call from channel application needs to pass through three or four different layers (not number of APIs) before it can return a response to the consumer. In case the second layer has aggregation and orchestration capability as outlined above, the call chain becomes really complex. Further complexity is added when an API in a specific layer calls another API from the same layer. For example, a second layer API calling another second layer API. This results in an overloaded API environment that grows exponentially and significantly increasing operations and development cost to simply expose a business capability to consumers.

How do we simplify this?

Content above may seem to suggest that layering is bad in APIs. However, that's not the case. Layering is important to provide a level of abstraction to the consumers. However, layering also needs to evolve with evolving architectural and technology patterns. Not all organisations I have worked with got it wrong :-). Many of them have implemented layers very efficiently that provides them agility, scalability and low cost of development & maintenance. In my opinion, organisations should look at following set of principles to achieve truly great API implementation.

• Treat APIs as a mechanism to expose a business capability and not a mechanism to create a business capability. A business capability should be created outside the API implementation paradigm and then this capability should be exposed as API using REST API implementation patterns.
• Avoid creating API layers for the sake of creating abstraction. While layering is important, creating APIs for the sake of creating abstraction is not a good practice. Organisations should challenge any API created in any layer to make sure it add business value and doesn't just create an abstraction for the sake of it. For example, do you really need a microservice wrapper on a Cloudant database that already provides REST API access to the database natively?
• Use best available technology to implement APIs and expose it via centralised API platform. Best implementation of an API layer may not always be microservice that's coded using industry best framework. Organisations must have flexibility in implementation for different use cases. For example, if you really need to expose resources managed in MongoDB using REST API, you may want to look at technologies like Eve rather than building this yourself. You should still expose Eve APIs via a centralised gateway to ensure consumers are not affected by the choice of technology underneath the API interface. This approach, however, opens up another challenge around standardisation. As different teams decide best possible technology to implement their APIs, the environment becomes difficult to manage due to its heterogenous nature. This can be avoided by standardising on a set of approved patterns and technologies to limit this proliferation.
• Not every API needs to be REST API. While you need APIs to access any business or technical capability, it doesn't necessarily needs to be a REST API. APIs take many forms. A java library like Hibernate provides a great layer of abstraction for different database without being a REST API. These can be catalogued centrally for team to access. This provides required layer of abstraction and reuse without getting trapped in REST APIfication of everything.

To summarise, with growing landscape and maturing technologies, organisations need to re-look at their API implementations to ensure they are creating scalable, manageable and cost effective API platform that can deliver benefits to the organisation.