Cloud Chronicles: (Part 2 of 3) Hybrid Cloud & Enterprise Application Integration (EAI) Patterns & Practices
Ryan Johnson
Senior Engineering Manager (Fullstack) | Cloud & Data Architect | Leader in Digital Innovation & Analytics
An Architects Guide to Securely Linking On-Prem Datacenters to the Cloud & the Enterprise Application Integration (EAI) Patterns for Achieving Enterprise-Grade Comms (Part 2 of 3)
This article is structured into three sections, investigating the fusion of on-premises data centers with cloud services, hybrid cloud integration patterns, reflections from the author, and key takeaways and lessons learned from the underlying project. The initial sections delve into integration principles, the influence of human nature on design, security measures for on-premises data centers, and the establishment of secure connections into the cloud. Parts 1 and 2 particularly concentrate on crucial considerations for Enterprise Application Integration (EAI) solutions, offering guidance for IT professionals and addressing common challenges, especially for cloud and solution architects navigating integration complexities stemming from hybrid hosting scenarios. The article attempts to provide best practices for leveraging (both brokered real-time and batched) messaging / event processing scenarios; highlighting the ongoing importance of continuous testing, documentation, and adaptability to evolving security landscapes. These sections lay the groundwork for 2 subsequent discussions from a post-mortem perspective---concluding with Part 3 which presents insights / lessons derived from the project forming the basis of this discussion (covering aspects such as layered security, verifiability, observability, durability, repeatability, and performance and extending to meeting requirements related to availability, latency, operations per second, throughput, resiliency, authentication and authorization, telemetry, logging, and recoverability).
Project Recap
Serving as both a solution architect and technical delivery lead for one of the world's largest telecommunications organizations, my fellow senior IT professionals and I led our teams in developing an automated integration architecture. This architecture connected Salesforce and Microsoft SharePoint Application frameworks, serving as a highly enriched LOB (on-premises) data repository. It facilitated document management, workflow processing, and contracting quotes among other transactions to streamline the sales cycle. These lessons should inform best practices and design patterns for creating data integration fabrics and services using disparate data stores. They pass both real-time and brokered messaging along using various modern data structures to ensure 1:1 fidelity of both transactional and canonical reference data. These lessons aim to aid others undertaking similar tasks. While the technologies used are fascinating, the following elements are not the primary focus of this discussion and will receive only brief attention: IAM/AD, ADFS, OAuth, RBAC, Inheritance Modeling, MS SQL Server, PowerShell, SSIS, Batch Frameworks, WebSocket, Event Services, and Analytics Services.
?
Summary of Lessons Learned
This section outlines key insights and guidelines for technology implementation and Enterprise Application Integration (EAI) selection. It emphasizes the importance of middleware in integrating on-prem applications hosted in your private data center to cloud managed solutions (SharePoint with Salesforce for this example) and suggests optimal solutions for real-time processing and scalable site provisioning. In this discussion I try to provide selection criteria for real-time integration, bulk processing, and low latency services, advocating for a careful approach to avoid real-time integration and promote decoupling to prevent system issues. It stresses the importance of understanding the differences between REST and SOAP, utilizing Azure Messaging for communication, and adhering to ACID and SOLID principles for system reliability and maintainability.
·??????? Key Insights from Technology Implementation:
o?? Middleware is essential for integrating SharePoint farms and Salesforce, facilitating extensive messaging.
o?? For robust document repositories and workflow engines combining SharePoint and Salesforce, avoid real-time integrations and breaking permission inheritance models for smoother operations.
·??????? EAI Selection Guidelines:
o?? Batch applications should utilize QUEUES for brokered messaging, and low latency services should opt for WCF/SOAP/REST/WebSockets.
o?? REST is ideal for stateless CRUD operations, but orchestration may still require WCF/SOAP, and WebSockets are excellent for complex bi-directional operations.
·??????? Middleware Messaging Strategies:
o?? Avoid real-time integration when possible and prioritize decoupling to prevent race conditions and deadlocks.
o?? Understand the limitations and strengths of REST and SOAP and consider data integration engines and tooling.
?
Assessing Enterprise Application Integration Patterns and Practices
Bus/hub
This is usually implemented by enhancing standard middleware products (application server, message bus) or as a stand-alone program (i.e., does not use any middleware), acting as its own middleware.
Application connectivity
The bus/hub connects to applications through a set of adapters (also referred to as connectors). These programs know how to interact with an underlying business application. The adapter performs two-way communication, carrying out requests from the hub against the application and notifying the hub when an event of interest occurs in the application (e.g., a new record inserted, a transaction completed, etc.). Adapters can be specific to an application (e.g., built against the application vendor's client libraries) or specific to a class of applications (e.g., can interact with any application through a standard communication protocol, such as SOAP, SMTP, or Action Message Format (AMF)). The adapter could reside in the same process space as the bus/hub or execute in a remote location and interact with the hub/bus through industry-standard protocols such as message queues, web services, or even use a proprietary protocol. In the Java world, standards such as JCA allow adapters to be created in a vendor-neutral manner.
Data format and transformation
To avoid every adapter having to convert data to/from every other application's format, EAI systems usually stipulate an application-independent (or common) data format. The EAI system usually provides a data transformation service as well to help convert between application-specific and common formats. This is done in two steps: the adapter converts information from the application's format to the bus's common format. Then, semantic transformations are applied to this (e.g., converting zip codes to city names, splitting/merging objects from one application into objects in other applications, etc.).
Integration modules
An EAI system could be participating in multiple concurrent integration operations at any given time, each type of integration being processed by a different integration module. Integration modules subscribe to events of specific types and process notifications that they receive when these events occur. These modules could be implemented in different ways: on Java-based EAI systems, these could be web applications or EJBs or even POJOs that conform to the EAI system's specifications.
?
Support for transactions
When used for process integration, the EAI system also provides transactional consistency across applications by executing all integration operations across all applications in a single overarching distributed transaction (using two-phase commit protocols or compensating transactions).
Brokered Messaging amp; Queues
These can be thought of as asynchronous, or decoupled messaging features that support publish-subscribe, temporal decoupling, and load balancing scenarios using the Service Bus messaging infrastructure. Decoupled communication has many advantages; for example, clients and servers can connect as needed and perform their operations in an asynchronous fashion.
Queues offer First In, First Out (FIFO) message delivery to one or more competing consumers. FIFO means that messages are typically expected to be received and processed by the receivers in the temporal order in which they were enqueued, and each message will be received and processed by only one message consumer. A key benefit of using queues is to achieve temporal decoupling of application components: in other words, the producers and consumers do not need to be sending and receiving messages at the same time, since messages are stored durably in the queue. A related benefit is load leveling, which enables producers and consumers to send and receive messages at different rates.
Service-Oriented Architectures (SOA)
An architectural style whose goal is to achieve loose coupling among interacting software agents / services.
Service-Oriented Architecture, or SOA, enables IT departments to make the transition from an application-centric view of the world to a process-centric view. Today, IT departments have the freedom to combine business services from multiple applications to deliver true end-to-end support for business processes. And, because the integration mechanism of SOA (usually Web Services) enables loosely coupled integration, IT departments can upgrade or change applications without impacting other applications.
Though SOA is being increasingly implemented both as green field (top down) and legacy modernization (bottom up), there is a clear lack of testing methodologies designed specifically for SOA applications. New approaches and methodologies are necessary to verify and validate applications based on SOA concepts.
The following key principles are recommended when implementing SOA:
???????? Document the Business Processes. Be it bottom up or top down, availability of these Business.
???????? Process documentation is critical in delivering SOA in its true self instead of saying, Web Services, based applications.
???????? SOA Implementation is an evolution – start with a pilot, deliver business value, and incrementally add on
???????? The SOA Implementation must be based on loosely coupled services that provide the highest flexibility and ongoing cost reduction due to reuse and lower maintenance.
???????? The services should have standards compliant interfaces to enable seamless integration and interoperability with other services.
???????? The business drives the services, and the services drive the technology.
???????? Business agility is a fundamental to SOA.
Web Services
SOAP
??????????????????? SOAP stands for Simple Object Access Protocol
??????????????????? SOAP is an application communication protocol.
??????????????????? SOAP is a format for sending and receiving messages.
??????????????????? SOAP is platform independent.
??????????????????? SOAP is based on XML.
Relevance
??????????????????? It is important for web applications to be able to communicate over the Internet.
??????????????????? The best way to communicate between applications is over HTTP because HTTP is supported by all Internet browsers and servers. SOAP was created to accomplish this.
??????????????????? SOAP provides a way to communicate between applications running on different operating systems, with different technologies and programming languages.
SOAP Building Blocks
??????????????????? A SOAP message is an ordinary XML document containing the following elements:
??????????????????? An Envelope element that identifies the XML document as a SOAP message
??????????????????? A Header element that contains header information
??????????????????? A Body element that contains call and response information
??????????????????? A Fault element containing errors and status information.
?
SOAP 1.2
SOAP is fairly mature and well-defined and comes with a complete specification. The REST approach is just that, an approach and is wide open for development, so if you have the following then SOAP is a great solution:
??????????????????? Asynchronous processing and invocation; if your application needs a guaranteed level of reliability and security then SOAP 1.2 offers additional standards to ensure this type of operation. Things like WSRM – WS-Reliable Messaging.
??????????????????? Formal contracts: if both sides (provider and consumer) must agree on the exchange format then SOAP 1.2 gives the rigid specifications for this type of interaction.
??????????????????? Stateful operations; if the application needs contextual information and conversational state management then SOAP 1.2 has the additional specification in the WS* structure to support those things (Security, Transactions, Coordination, etc.). Comparatively, the REST approach would make the developers build this custom plumbing.
??????????????????? SOAP is still offered by some very prominent tech companies for their APIs (Salesforce, PayPal, DocuSign). One of the main reasons: legacy system support. If you built a connector between your application and Salesforce back in the day, there’s a decent probability that connection was built in SOAP.
There are a few additional situations:
??????????????????? SOAP is good for applications that require formal contracts between the API and consumer since it can enforce the use of formal contracts by using WSDL (Web Services Description Language).
??????????????????? Additionally, SOAP has built-in WS-Reliable messaging to increase security in asynchronous execution and processing.
??????????????????? Finally, SOAP has built-in stateful operations. REST is naturally stateless, but SOAP is designed to support conversational state management.
??????????????????? Some would argue that because of these features, as well as support for WS_AtomicTransaction and WS_Security, SOAP can benefit developers when there is a high need for transactional reliability.
REST
REST applications do not need a service contract a la SOAP (WSDL), So this means areas that REST works well for are:
???????? Limited bandwidth and resources; remember the return structure is really in any format (developer defined). Plus, any browser can be used because the REST approach uses the standard GET, PUT, POST, and DELETE verbs. Again, remember that REST can also use the XMLHttpRequest object that most modern browsers support today, which adds an extra bonus of AJAX.
???????? Totally stateless operations: if an operation needs to be continued, then REST is not the best approach and SOAP may fit it better. However, if you need stateless CRUD (Create, Read, Update, and Delete) operations, then REST is it.
领英推荐
???????? Caching situations: if the information can be cached because of the totally stateless operation of the REST approach, this is perfect.
REST over HTTP is an appealing option. It is simple to use and understand and requires no heavyweight framework or toolchain to get started. It lends itself well to testing and reduces many operational concerns to the same practices used to manage.
?
Use Cases for REST
Case 1: Developing a Public API
REST focuses on resource-based (or data-based) operations and inherits its operations (GET, PUT, POST, DELETE) from HTTP. This makes it easy for both developers and web-browsers to consume it, which is beneficial for public APIs where you don’t have control over what’s going on with the consumer. Simplicity is one of the strongest reasons that major companies like Amazon and Google are moving their APIs from SOAP to REST.
Case 2: Extensive Back-and-Forth Object Information
APIs used by apps that require a lot of back-and-forth messaging should always use REST. For example, mobile applications. If a user attempts to upload something to a mobile app (say, an image to Instagram) and loses reception, REST allows the process to be retried without major interruption once the user regains cell service.
However, with SOAP stateful operations, the same type of service would require more initialization and state code. Because REST is stateless, the client context is not stored on the server between requests, giving REST services the ability to be retried independently of one another.
Case 3: Your API Requires Quick Developer Response
REST allows easy, quick calls to a URL for fast return responses. The difference between SOAP and REST, in this case, is complexity—-SOAP services require maintaining an open stateful connection with a complex client. REST, in contrast, enables requests that are completely independent of each other. The result is that testing with REST is much simpler.
Helpfully, REST services are now well-supported by tooling. The available tools and browser extensions make testing REST services continually easier and faster.
Messaging
Service Bus
Whether an application or service runs in the cloud or on premises, it often needs to interact with other applications or services. To provide a broadly useful way to do this, Microsoft Azure offers Service Bus.?
Service Bus fundamentals
Different situations call for different styles of communication. Sometimes, letting applications send and receive messages through a simple queue is the best solution. In other situations, an ordinary queue isn't enough; a queue with a publish and-subscribe mechanism is better. In some cases, all that's really needed is a connection between applications; queues aren't required. Service Bus provides all three options, enabling your applications to interact in several different ways.
Within a namespace, you can use one or more instances of four different communication mechanisms, each of which connects applications in a different way. The choices are:
???????? Queues, which allow one-directional communication. Each queue acts as an intermediary (sometimes called a broker) that stores sent messages until they are received. Each message is received by a single recipient.
???????? Topics, which provide one-directional communication using subscriptions-a single topic can have multiple subscriptions. Like a queue, a topic acts as a broker, but each subscription can optionally use a filter to receive only messages that match specific criteria.
???????? Relays, which provide bi-directional communication. Unlike queues and topics, a relay doesn't store in-flight messages; it's not a broker. Instead, it just passes them on to the destination application.
Queues
A sender sends a message to a Service Bus queue, and a receiver picks up that message at some later time. A queue can have just a single receiver. Or multiple applications can read from the same queue. In the latter situation, each message is read by just one receiver. For a multi-cast service, you should use a topic instead.
A receiver can read a message from a Service Bus queue in two different ways. The first option, called ReceiveAndDelete, removes a message from the queue and immediately deletes it. This is simple, but if the receiver crashes before it finishes processing the message, the message will be lost. Because it's been removed from the queue, no other receiver can access it.
?
The second option, PeekLock, is meant to help with this problem. Like ReceiveAndDelete, a PeekLock read removes a message from the queue. It doesn't delete the message, however. Instead, it locks the message, making it invisible to other receivers, then waits for one of three events:
???????? If the receiver processes the message successfully, it calls Complete, and the queue deletes the message.
???????? If the receiver decides that it can't process the message successfully, it calls Abandon. The queue then removes the lock from the message and makes it available to other receivers.
???????? If the receiver calls neither of these within a configurable period (by default, 60 seconds), the queue assumes the receiver has failed. In this case, it behaves as if the receiver had called Abandon, making the message available to other receivers.
Queues are useful in quite a few situations. They enable applications to communicate even when both aren't running at the same time, something that's especially handy with batch and mobile applications. A queue with multiple receivers also provides automatic load balancing, since sent messages are spread across these receivers.
Topics
Useful as they are, queues aren't always the right solution. Sometimes, Service Bus topics are better.
A topic is similar in many ways to a queue. Senders submit messages to a topic in the same way that they submit messages to a queue, and those messages look the same as with queues. The big difference is that topics enable each receiving application to create its own subscription by defining a filter. A subscriber will then see only the messages that match that filter. For example this figure shows a sender and a topic with three subscribers, each with its own filter:
???????? Subscriber 1 receives only messages that contain the property Seller="Ava".
???????? Subscriber 2 receives messages that contain the property Seller="Ruby" and/or contain an Amount property whose value is greater than 100,000. Perhaps Ruby is the sales manager, so she wants to see both her own sales and all big sales regardless of who makes them.
???????? Subscriber 3 has set its filter to True, which means that it receives all messages. For example, this application might be responsible for maintaining an audit trail and therefore it needs to see all the messages.
Relay
Both queues and topics provide one-way asynchronous communication through a broker. Traffic flows in just one direction, and there's no direct connection between senders and receivers. But what if you don't want this? Suppose your applications need to both send and receive messages, or perhaps you want a direct link between them, and you don't need a broker to store messages. To address scenarios such as this, Service Bus provides relays, as this figure shows.
Service Bus relay provides synchronous, two-way communication between applications.
The obvious question to ask about relays is this: why would I use one? Even if I don't need queues, why make applications communicate via a cloud service rather than just interact directly? The answer is that talking directly can be harder than you might think. Suppose you want to connect two on-premises applications, both running inside corporate datacenters. Each of these applications sits behind a firewall, and each datacenter probably uses network address translation (NAT). The firewall blocks incoming data on all but a few ports, and NAT implies that the machine each application is running on doesn't have a fixed IP address that you can reach directly from outside the datacenter. Without some extra help, connecting these applications over the public internet is problematic.
A Service Bus relay can help. To communicate bi-directionally through a relay, each application establishes an outbound
TCP connection with Service Bus, then keeps it open. All communication between the two applications travels over these connections. Because each connection was established from inside the datacenter, the firewall allows incoming traffic to each application without opening new ports. This approach also gets around the NAT problem, because each application has a consistent endpoint in the cloud throughout the communication. By exchanging data through the relay, the applications can avoid the problems that would otherwise make communication difficult.
To use Service Bus relays, applications rely on the Windows Communication Foundation (WCF). Service Bus provides WCF bindings that make it straightforward for Windows applications to interact via relays. Applications that already use WCF can typically just specify one of these bindings, then talk to each other through a relay.
Unlike queues and topics, applications don't explicitly create relays. Instead, when an application that wishes to receive messages establishes a TCP connection with Service Bus, a relay is created automatically. When the connection is dropped, the relay is deleted. To enable an application to find the relay created by a specific listener, Service Bus provides a registry that enables applications to locate a specific relay by name.
Relays are the right solution when you need direct communication between applications. For example, consider an airline reservation system running in an on-premises datacenter that must be accessed from check-in kiosks, mobile devices, and other computers. Applications running on all these systems could rely on Service Bus relays in the cloud to communicate, wherever they might be running.
Centralized Relay Service
Azure Service Bus provides two comprehensive messaging solutions – one, through a centralized "relay" service running in the cloud that supports a variety of different transport protocols and Web services standards, including SOAP, WS-*, and REST. The client does not need a direct connection to the on-premises service, nor does it need to know where the service resides, and the on-premises service does not need any inbound ports open on the firewall.
Testing Service-Oriented Architectures (SOA)
The V-Model is a suitable test methodology to deliver SOA projects for the following reasons:
???????? It encourages a top-down approach with respect to defining the business process requirements, high-level functional technical design, security etc.
???????? It then encourages a bottom-up test approach – test individual functions within a service, test an individual service then test a composite set of services through to testing an integrated process and finally testing a complete business system. SOA is 'loosely' coupled services and that is why a bottom-up test approach is recommended.
???????? The levels reflect different viewpoints of testing on different levels of detail.
The V-model encourages testing throughout the entire Software Development Life Cycle.
You can categorize SOA testing into the following phases:
???????? Governance Testing
???????? Service-component-level testing
???????? Service-level testing
???????? Integration-level testing
???????? Process/Orchestration-level testing
???????? System-level testing
???????? Security Testing
???????? Governance Testing
SOA Governance is a key factor in the success of any SOA Implementation. It is also the most 'loosely' used term, as it covers the entire lifecycle of SOA Implementation – from design to run time to ongoing maintenance. SOA Governance refers to the Standards and Policies that govern the design, build and implementation of a SOA solution and the Policies that must be enforced during runtime.
In the next post, we will examine more EAI patterns for security, messaging, and performance optimization. Cheers!