Learnings and observations from implementing contract testing

Recently, I've been working to put together a proof of concept for adding contract tests to our automated testing repertoire here at Banked. It's a tool that I'd heard of and discussed regularly over the last few years, but never had the opportunity to build from scratch myself, so when the opportunity arose here, I was excited to take it on.

What is contract testing, and why choose it?

For those that don't know, contract testing is a type of testing (frequently mentioned alongside Pact ) that mainly focusses on the integration and communication between different services. As a lot of organisations have transitioned to a micro-services architecture, we've come across issues where Team A will make a change in Service A, that has unexpected impacts on a consumer of that service, Service B, owned by a completely different team, who were unaware this change was happening, let alone that it would also impact them. Suddenly alerts are firing, customers are impacted, and stakeholders are frustrated. Contract testing aims to help teams be more aware of their dependencies and to talk about their changes before problems are hitting production.

In any given contract, you have two parties: the consumer, who will be making a request to an API and expecting a specific response; and the provider, who provides the API and will be responding to the request. The contract then contains all the expected interactions from the consumer's perspective, and the provider will then have all these interactions replayed against it to validate that it responds as expected. If, when testing against the provider, we find a discrepancy between what's expected in the contract and what the provider actually responds with, then we know some changes are needed somewhere in order to get these two services aligned again.

At Banked, our architecture fits in with what I described earlier: A number of micro-services, all owned by various different teams, and written in multiple different languages. As we've rapidly grown our engineering teams and the product itself, it's become more challenging to be aware of all of the potential impacts a change could introduce. With that in mind, we felt like contract testing would help provide some additional confidence when we want to merge in changes.

Getting our POC up and running

Our services are mainly built in either Go, Ruby, or Javascript/Typescript, all languages that Pact has libraries for (You can see the full list in the side panel here ), and I identified two Go services that had a single interaction between them; Perfect to use for a POC with pact-go .

As a sidebar, pact-go is currently undergoing quite a lot of changes from it's main branch to the 2.x.x branch, and I initially had a lot of challenges getting the main branch version to work on my M1 Mac, and never quite figured out how to get around them. So if you're working on an ARM machine like me, I'd recommend using the 2.x.x version as well.

Pact tests follow a relatively simple flow:

• Provide config for your mock provider
• Create any request/response body matchers that you might need
• Write the Pact test itself, setting out your expectations and calling the method that makes the API request

Pact then handles intercepting the actual request and returning the response you specify.

Consumer Testing

In my example below, I'm looking to validate a request made from our refunds-service, the consumer, to our mas-rest-api service, the provider. It's a POST request, with a JSON body in the request and the response. With that example in mind, lets take a look at how the above steps play out:

mockProvider, err := consumer.NewV2Pact(consumer.MockHTTPProviderConfig{
   Consumer: "refunds-services",
   Provider: "mas-rest-api",
   Host:     "127.0.0.1",
   Port:     55532,
})        

First up, we have our mock provider config. Adding the consumer and provider names will allow the provider half of the tests to know which contracts relate to it. I've added a specific host and port just so I know where exactly the mock can be located.

requestBodyMatcher := Like(map[string]interface{}{
   "externalReferenceId": Like("test-refund-1000"),
   "merchantAccountId":   UUID(),
   "amount": Like(map[string]interface{}{
      "amount":   Like("400"),
      "currency": Term("GBP", "GBP|EUR|PLN"),
   }),
   "reference": Like("Refund."),
   "individualBankAccount": Like(map[string]interface{}{
      "firstName": Like("Paw"),
      "lastName":  Like("Patrol"),
      "sortCodeAccountNumber": Like(map[string]interface{}{
         "sortCode":      Like("123456"),
         "accountNumber": Like("12345678"),
      }),
   }),
})

responseBodyMatcher := Like(map[string]interface{}{
   "payout": Like(map[string]interface{}{
      "id":                  UUID(),
      "externalReferenceId": Like("test-refund-1000"),
      "merchantAccountId":   UUID(),
      "amount": Like(map[string]interface{}{
         "amount":   Like("400"),
         "currency": Term("GBP", "GBP|EUR|PLN"),
      }),
      "reference": Like("Refund."),
      "status":    Like("PAYOUT_STATUS_PENDING"),
      "createdAt": Like("2023-01-25T13:32:30.791103985Z"),
      "updatedAt": Like("2023-01-25T13:32:30.791103985Z"),
   }),
})        

After declaring our mock provider, we've then got our two body matchers for the request and response. Pact provides several different matcher functions that allow you to match on various types. For example, for all our different IDs we use the UUID() matcher, and whenever we have fields that have specific set values, like currency, we can use the Term() matcher and a regex to specify the accepted values.

err = mockProvider.
   AddInteraction().
   Given("We want to create a payout").
   UponReceiving("A valid request").
   WithRequest("POST", "/v1/payouts", func(b *consumer.V2RequestBuilder) {
      b.JSONBody(requestBodyMatcher)
   }).
   WillRespondWith(201, func(b *consumer.V2ResponseBuilder) {
      b.JSONBody(responseBodyMatcher)
      b.Header("Content-Type", S("application/json;charset=UTF-8"))
   }).
   ExecuteTest(t, func(config consumer.MockServerConfig) error {
      client := mas.New("https://127.0.0.1:55532")

      _, err := client.CreatePayout(context.TODO(), createPayoutRequestObject)

      assert.NoError(t, err)

      return err
   })

assert.NoError(t, err)        

Finally, we specify our actual test. Pact is quite legible with a BDD-like syntax. The WithRequest() method outlines what we expect our consumer to send to the provider, and the WillRespondWith() method outlines our expectations for the providers response. In both of these, we make use of the matcher objects we created above.

Lastly, the ExecuteTest() method is what we use to actually call the HTTP method itself. In my test, I've not been too bothered about what comes back from that method currently, but we could do some additional assertions here against the return value of the method if we wanted to. For this, I just wanted to validate that we've gotten our successful response and handled it without error.

Running this test will generate the contract, containing all the information required in order to replay the same request against our actual provider to ensure it responds as expected.

Using a Pact Broker

It's not specifically required to make use of contract testing, but within this context, a Pact Broker is basically an intermediary service that provides the ability to share contracts amongst your different services and to store validation data. You can self-host Pact Broker using the supplied Docker image , or you can also use the SaaS version called PactFlow . As it's so easy to get up and running with, and prevents me from needing to manually pass around the contract files myself, I decided to use PactFlow's free tier for our Pact Broker.

Some of the language specific Pact libraries provide built-in functionality for publishing contracts to the Pact Broker, but since the pact-go one does not, I needed to make use of the pact-cli Docker image to do so. Once the contract is published, the consumer half of the testing is complete.

Testing our Provider

The last step in all of this is to validate our contract against the provider. The code for this is incredibly simple:

err := verifier.VerifyProvider(t, provider.VerifyRequest{
   ProviderBaseURL: "https://mas-rest-api",
   Provider:        "mas-rest-api",
   BrokerURL:       os.Getenv("PACTFLOW_URL"),
   BrokerToken:     os.Getenv("PACTFLOW_AUTH_TOKEN"),
   ConsumerVersionSelectors: []provider.Selector{
      &provider.ConsumerVersionSelector{
         Branch: "main",
         Latest: true,
      },
   },
   PublishVerificationResults: true,
   ProviderVersion:            os.Getenv("VERSION"),
})

assert.NoError(t, err)        

So there's a number of values here we've needed to use. Most of them are pretty self explanatory, but let's go through them all.

• ProviderBaseURL is the actual base URL we want to send our requests to. The endpoint I'm calling is only available in our internal Kubernetes cluster, so I'm using a relevant address for that. When writing contract tests, it's worth considering whether things besides the service itself can amend the request or response, and how you want to handle that.
• Provider is how Pact knows what contracts relate to this test, and it needs to match what we specified in our consumer test.
• BrokerURL and BrokerToken relate to your Pact Broker. If you're not using a Broker, you'd want to make use of the PactFiles field to provide a file path instead.
• ConsumerVersionSelectors provides you with different ways for selecting which contracts you want to validate against. Pact gives you the ability to test against contracts generated on branches other than main so that teams can make changes in tandem, and updating your ConsumerVersionSelectors is how you can point your provider to those other contracts.
• PublishVerificationResults will ensure that these results are published back in the Pact broker once the test has run.

Having run the provider test, we now know whether our services are aligned, and if we have a failure, we know a conversation needs to happen.

Challenges and learnings

Getting the tests themselves running locally wasn't too difficult, but I ended up scratching my head a few times when trying to get this up and running in our CI. Here are a couple of issues and observations I made throughout the POC:

1. Where am I actually running these tests? Running tests for the consumer seemed relatively straightforward. We have a bunch of CI workflows in Github Actions for the service, so I added an extra step that was triggered on PR and merge into main that would run the tests and publish the contract. In an ideal world, the provider tests would then run on PR as well so that we could learn about any issues before code gets merged in. Unfortunately, because most of our services have dependencies on other services, and we don't currently want to maintain a bunch of mocks for things like this, there isn't a straightforward and simple way for us to be able to run the provider tests within our PR validations. To get around this for the time being, we have the tests run as a post-sync job once the service has successfully been deployed into our dev environment, but longer term we want to come up with a solution to have temporary environments that we can quickly scale up and down for testing things like this.
2. Why am I getting an unexpected response back from the provider? One thing that I've raised an issue for in the pact-go repo is that there isn't really a way to debug a failed test on the provider side. For example, at one point I was running the test and getting back a 422 from the provider instead of a 201. All that Pact reports however is that the status and response body doesn't match what's in the contract. If I'd been able to see the response body, I would have realised that one of the IDs I was providing in my request was a randomly generated one by Pact instead of the specific one I needed, but it needed me to go back the consumer service and debug in order to figure out the issue. Unfortunately I don't have a work around for this, it's just not a great dev experience, but I've got my fingers crossed that the Pact team will address this.
3. It would have been excellent if we'd started with Pact from Day 1! I think this goes without saying for all test automation, but it feels notably more painful to have to retrofit tests onto production code than it does to write tests alongside your code. Whilst we might not necessarily want to use Pact on every service, I'm soon going to have the challenge of trying to ensure our dev teams actually take up Pact and make use of it, which means battling with other competing priorities. Definitely a tool that I think benefits from starting with early.
4. Quality of documentation, and the Pact Specification. A couple of final points all rolled into one. The Pact docs are generally pretty decent, but it relies a lot on the Github docs for the libraries themselves, and Github documentation is simply never going to be as in-depth and detailed as specific dev documentation for a tool. I compare it to the Cypress documentation for example, and the difference is notable. I think it's understandable considering it's a tool with libraries in multiple different languages, but it's something to be aware of. Pact itself also has a specification that all the different libraries can use to ensure parity between matchers. They're currently on Version 4 of that specification, but older versions going back to V2 are still supported, and I personally found it wildly confusing at first trying to decipher whether I needed to use V2, V3 or V4, and what on earth the differences were meant to be. It's a lot less important if both services are in the same language, but if you're comparing against services in two different languages then you'll want to make sure that both libraries support the same version of the specification you want to use. The majority support V2, and are introducing support for V3 and V4.
5. Asking questions. Lastly, Pact have their own Slack community that you can join. In theory this is great, but in practice I found it was literally only the library maintainers themselves that responded to questions, and the maintainer for pact-go is based in Melbourne so I'm typically looking at a 24hr response time unless he happens to be looking at Slack after work hours. This might not be the case for all of the libraries, I think the Go library probably just doesn't have as big a community as Java or JS would, but if you're utilising Go as well, that's something else to keep in mind.

So that's a quick round up of putting together our proof of concept with Pact, and why it feels like it's going to be a great tool for us. I'm looking forward to expanding on the proof of concept further and seeing the impact it can have in our company. If you have any questions about Pact, feel free to drop me a message on here or find me in the Pact Slack.