EBA Annual Maintenance 2023: A Tale of Flags

Red Flags

In the fast-paced world of 24/7 payment processing, maintaining uptime while implementing crucial updates is a monumental challenge. Phased deployment of new features is one thing - but what about a Europe-wide upgrade of all interbank message formats in one weekend? When you exist between financial institutions and payment authorities, your responsibilities are an order of magnitude greater. This is real world impact: an outage prevents friends from splitting the dinner bill, a settlement delay causes missed direct debits, and any issues erode brand image.

At the heart of our engineering philosophy lies a commitment to continuous improvement. When faced with the daunting task of EBA Annual Maintenance 2023, our team saw it not just as an opportunity to shield our customers from complexity but as a chance to refine our processes and enhance our payment platform's capabilities.

The approach of the old world would be to take our systems offline at the start of the maintenance window, frantically deploy new versions, and ensure everything was operational before opening the floodgates. This is the approach countless institutions across Europe took. Did it work? Mostly. Is it a good end-user experience? No.

So what happens when you challenge a group of talented engineers to do better? In this case, you get time-gated feature flags.

Feature Flags

A feature flag, also known as a feature toggle or feature switch, is a technique which allows developers to turn certain "features" (think functionality) of a software application on or off dynamically, without deploying new code. Feature flags are typically controlled through configuration settings such as environment variables, allowing developers to enable or disable specific features remotely.

This is a fantastic approach for phased deployment, A/B testing, emergency rollbacks, and easy cleanup once the feature is enabled everywhere by default. But what if you have a strict migration timeline for multiple services, with the cutover windows happening at unsociable hours? What if your feature is a completely different format for bulks of payment messages? What if you have to roll back to the old format for pending payments?

Turns out the answer is obvious when you have a critical mass of innovators.

1. Toggle the feature based on clock time. This avoids bleary-eyed configuration changes on the vampire shift.
2. Ensure you can change, or completely disable, the toggle times. Maintenance windows can contract and expand - code defensive logic for factors you cannot control.
3. Build both formats and submit whatever your feature flag mandates. This allows rollback and resubmission in the old format without manual intervention.

These solutions aren't specific to the world of payments. It can clearly work for any contract change, whether that's a small tweak to your event schema or a continent-wide shift.

Country Flags

Our engineers hail from all across the world. This gives us a powerful blend of opinions - and some good cultural stories to exchange while we let the computers do the hard work (fun fact: most of our team consider p?czki to be proof of a higher power).

The hallmark of our approach to software development is empowering our engineers to innovate and take full ownership of their solutions. Unleashing this creativity and problem-solving prowess was instrumental in devising a solution which not only met technical requirements but also allowed our customers - and their customers - to operate as normal.

We had zero downtime, allowing end-users to continue managing their payments, blissfully unaware of the chaos unfolding across Europe. We also had zero payment impact despite leaving our systems online for the entire weekend.

Surprising? Not when you know the talented ladies and gentlemen working on our solutions. I continue to stand on the shoulders of giants and the view is spectacular.