Testable Integration Architecture

Testable Integration Architecture

Written by Tom Keeber , Principal Software Engineer

Hang around me long enough and I’ll probably start recommending books.

One of my favourites is?Enterprise Integration Patterns . Although this book is now close to 20 years old and contains scary words like “Enterprise” its still extremely relevant and the patterns it describes are a foundation for message based asynchronous systems.

Near the beginning of that book there is a great quote.

Change is inevitable

Change is inevitable you say? Well, at Dunelm we want to be able to move safely at speed. Which means we invest in our automated testing framework. Why do we do this? Because we want to make changes quickly and safely and if something is broken we want that feedback quickly.

In this blog post I’m going to talk about how the different architectural integration choices you make affect your ability to make changes (the extensible architecture side) and make them safely (the testing part).

Note; I’m aware that there are far more factors to be considered when creating software — I’m going to use the usual blog post get out clause — that they are out of the scope of this article.

There are 4 “Architectural Styles” for passing data between different systems defined in Enterprise Integration Patterns.

• File Transfer
• Shared Database
• Remote Procedure Invocation
• Messaging

Each of these integration options comes with their different challenges when it comes to testing. Lets talk through each…

File Transfer

If you’ve been in software long enough sooner or later you’ll be passing data around using File Transfer. Someone writes a file and someone reads it.

You often see File Transfer integration used for large data sets or where documents must be supplied in a pre-determined format.

Unfortunately, you also see File Transfer integration used where other integration mechanisms would do a much better job. This is usually older, 3rd party systems.

The big advantage here is the loose coupling between different systems. The Producer (creator of the file) and Consumer (reader of the file) could be written in entirely different frameworks/languages.

As with all integrations there needs to be a shared agreement on the format of the file.

Testing File Transfer Integration

File Transfer integration can be hard to test due to a lack of visibility, complexity, environment variability, and security concerns. You may also be limited in your automation due to certain file types. It is the lack of visibility and feedback that can hamper the benefits of automated testing — particularly if anything changes in file format by either the consumer or producer.

The file transfer mechanism acts as a natural testing barrier. For the outbound testing, where you are creating or updating a file, an automated test that creates a file and validates its format would a natural happy path integration test.

Where automation testing becomes tricky is the End-to-end journey and error scenarios. As this type of integration depends so much on the underlying File system — network connectivity, server capacity, and other environmental variables are very tricky to automate the testing of.

Advantages

• Simplicity of integration
• Reliability of the File based systems.

Disadvantages

• No Feedback on success.
• Anything can happen once you’ve placed that file into a shared space.
• Little visibility of failures
• Delay — far slower than other integration methods.

Shared Database

One database or schema is shared amongst multiple systems.

Fast becoming one of my least favourite integration types and a massive anti-pattern in a microservice domain driven architecture.

The big advantage of this type of integration is the ability to share the data amongst systems and know that data is from a single source of truth.

But it does come with some big challenges. One of the big ones will be how you manage database connections and load on your database (read/write replicas and other strategies will help with this). The other, and the one that really effects testing, is the ability to understand how schema and data changes effect all the systems talking to the database. If any system can make a schema change to the database, then you have a big problem in ensuring integrity.

Shared Database Integration Testing

Unlike File Transfer integration — Shared database integration requires you to think about feedback you get from changes in a different way. If any system can modify the schema or data anywhere in the database then you will run into problems which may not be easily identified — until it breaks.

When using this pattern, follow these guidelines to make sure that you have observability of integrated systems and get feedback on data and DDL changes;

• All systems that use a shared database must have an integration test that covers the systems use of the database.
• If you have multiple repositories with systems that access the database then keep DDL changes deployed and managed in a separate repo. Make sure your CI/CD for this repo either covers all systems use of the database or starts pipelines which perform automated tests.
• Ideally use a real database and not an in-memory version.
• Make sure your database connection strategy covers your maximum usage scenario for scaling. You don’t want one system starving others of connections.

Advantages

• Single source of truth for data
• Data up to date.
• Can use standard database testing patterns.

Disadvantages

• Feedback on DDL or data changes across systems can be cumbersome.
• Performance testing could be difficult — ability to replicate multiple users of a system adds complexity.
• Handling Database connections requires an enforced strategy and could be hard to test for.

Remote Procedure Invocation

Remote Procedure Invocation?applies the principle of encapsulation to integrating applications.

This is where I think the Enterprise Integration Patterns book needs an update.

Instead of RPI — I think a more modern description would be?API based integration. The idea is that we introduce loose coupling between systems and data by providing a predefined way of performing operations on or retrieving that data.

There are number of different implementations this some older ones such as Java RMI and some more recent such as Google created?gRPC . Probably the most popular is one you will know well — Web Services or REST based services usually over HTTP.

This integration style covers a huge and popular area of integration between systems and has some major advantages — ease of use, lots of implementation options with popular and well defined standards.

One of the big disadvantages is the use of networking and HTTP can introduce unexpected failures that often need to be handled by the client.

So what does this mean for testing.

Well, if well designed we can think of our web service as black box. We shouldn’t need to understand the details of what the services is doing, but only the effects of that action.

At a high level its important that we design our services in such a way that;

• If we have requested a operation on data we can determine that operation was success. (E.g. I have a system that manages Orders. Once I’ve created a new Order can I now use a service to find that Order)

This can lead to a simple testing strategy. But it is often not enough particularly in large complex systems.

At Dunelm we use a lot of REST based micro services for us this means ideally each service should;

• Provide a defined contract and as simple an interface as possible
• Provide a way for a calling system to set up and tear down data that might be required for integration testing.
• If this is not possible then provide a mocking mechanism for integration testing with other systems.
• Using tools such as PACT to provide contract based testing.

Advantages

• Simple, well understood.
• Lots of implementation options and testing options.
• Lots of testing tools available — such as Postman.
• Quick feedback can be received if the contract is not adhered to

Disadvantages

• Emphasis on the client to handle failures
• Contract changes need to be handled very careful. Just like DDL changes in a Shared Database Integration pattern — without an adequate testing strategy strategy — feedback and visibility of changes could lead to broken systems and unhappy clients.

Messaging

Asynchronous communication solves so many problems in a complex distributed system. But also adds a huge amount of complexity to our systems particularly around testing.

Again it comes down to feedback — in a Asynchronous system — how long do you wait for something NOT to happen? and how do I get feedback that something?has?happened?

You need to decide on a few different strategies on how you are going to test.

• Message payload validation — all messages should conform to the expected format.
• Message routing and delivery: Asynchronous systems rely on message brokers or middleware to route messages between components. You need to test that messages are routed correctly, delivered to the intended recipients, and processed in the correct order.
• Message acknowledgement and retry: You need to test that the system is capable of acknowledging messages and retrying failed deliveries when necessary. This includes testing for scenarios where messages are lost, delayed, or delivered out of order.
• Error handling and recovery: Asynchronous systems need to be resilient to errors and failures. You need to test that the system is capable of handling errors gracefully, recovering from failures, and resuming normal operation as quickly as possible.

Couple of practicals on this;

• Never use Thread.sleep or any other hard coded waiting mechanism in your tests.
• Looks to use CountdownLatches and await while you are waiting for something to happen.
• Think about how do I know the message I have sent has performed the operation I need it too, or updated the system I want it too? How long am I going to wait for this to happen?

Advantages

• Asynchronous communication allowing for improved performance and use of resources.
• Resilience and message guarantee.

Disadvantages

• Testing can become more complex — specifically — Asynchronous communication means you are making a conscious choice to not get feedback as to whether an operation is successful.

Conclusion

An architecture that restricts automated testing will restrict business agility and ultimately lead to decrease in productivity.

If your goal is to create a system that is flexible and building for velocity, then choosing your integration option will massively effect your testing strategy and the cost of automated testing.

If we make the wrong choices with our integration architecture testing will be harder. The boundaries around our tests can become blurred, testing loops and feedback can become harder to determine.