The Pillars of Agile Integration

For those of you who have worked in Enterprise Integration over the past 20 years, you will have seen a plethora of technologies and innovations come and go. In the late 1990s, the traditional way of point-to-point messaging between applications was enriched with a more process driven view of integration. In the mid 2000s then, companies embraced a standards driven view of middleware with SOA and Web Services leading the way, and BPM breaking out core functions into a separate business process and business rules management platform.

The ever present promise of "loosely coupled" systems however never really materialised - customers ended up building large and complex integration platforms with an immutable Enterprise Service Bus at its core. The spaghetti code of point-to-point integration was finally gone, but the resulting infrastructure diagrams of integration platforms did not look less complicated.

The concept of SOA and Web Services dominated the airwaves over the next years. But the design of Web Services based solutions had to adhere to a very formalised framework and eventually become too complex, too inflexible and too expensive to implement and maintain. Also, the big challenge of SOA based solutions was that the IT departments implementing them were not able to articulate their business value since the business never had a skin in the game.

But the idea of integrating applications using fine grained and loosely coupled services served as a great foundation for developing a more agile, more flexible and more cost effective approach to integration.

And that is what we now call "Agile Integration" which consists of three pillars:

• Microservices
• Containers
• APIs

Microservices

Microservices are the natural evolution of the SOA type Web Services, except that that they are simpler and smaller, assuming that their endpoints are the source of truth and that the services themselves focus on exchanging, transforming and validating messages only. In this environment an overly complex Enterprise Service Bus as a mediation layer is not needed anymore.

The other important aspect of microservices is that they are focused on business capabilities rather than technical capabilities. This requires the business to first decompose their business into discrete functions and to own their services from that point on. The purpose of IT teams is then to effectively implement the microservices on the platform of their choice.

To reduce complexity, microservices need to be self contained and loosely coupled so that they can be deployed in a distributed environment. In order to maintain clear boundaries between the services and making them resilient and flexible, the approach is to adopt a simple way of synchronous communication such as REST that hides some of the implementation complexities.

Finally, despite the focus on the business capabilities of microservices, it is critical to look at where the data used in a microservice are coming from: are they directly retrieved from the endpoint application ? Or do they reside in a data services layer that provides an abstracted view of the data in an enterprise. Aligning microservices with an efficient data strategy is critical.

Containers 

Whereas microservices are concerned with business functions and capabilities, containers play an increasingly important role for the question where and how these microservices are deployed. Containers have exploded over the past years and more companies are seeing the benefits that a container based platform bring: a footprint that is much lighter than virtual machines, portability across environments and an efficient way to package applications and their dependencies.

A comprehensive container platform bridges the gap between development and operations team and is a tool that brings the sometimes overused concept of DevOps to life:

• IT Operations control the hardware, virtualisation layer and the operating system and ensure they are patched, secure and managed in accordance to best practises.
• Developers control the applications, associated frameworks and libraries used in the applications and package them as containers which are delivered to IT Operations. 

Finally, a Container Platform can provide all functions for an integrated Platform as a Service that can run on premise in a physical environment, in virtualised environments or in the public cloud or a private cloud. Software images for application and integration platforms, business process and business rules management systems as well as SQL or NoSQL databases provide a solid foundation for developers to build, deploy and test their microservices.

APIs

Finally, microservices encapsulating business capabilities running on a Container Platform are only valuable if and when they can be consumed securely via APIs by their target audience - be it external or internal users, or both.

This is where the winning combination of an API Gateway and an API Management Solution becomes indispensable. The API Gateway will provide physical access to the APIs, whereas the API Manager holds the profiles and policies defining access control to the APIs, rate limits and usage policies, as well as analysis of traffic patterns to track how the APIs are used.

End users need to be able to sign up for application plans and access the API documentation through an embedded Developer Portal. And if companies decide to charge their end customers for the usage of APIs it becomes critical that the API Management solution is fully integrated into a invoicing and payment solution.

Finally, whilst companies are moving towards an agile integration approach it is critical that existing middleware layers are being utilised as required. A big bang approach is seldom a good idea, so leveraging the existing middleware landscape whilst moving towards an API, microservices and container led integration platform would be the recommended approach.

Feel free to talk to me about how Red Hat can assist you on your journey towards agile integration.

Daniel Soffner

Senior Solution Architect, Cloud and Emerging Technologies

Red Hat Australia, Melbourne