It’s all about the data! Fragmentation in the Financial Enterprise.

We live in a Fragmented Universe. This fragmentation is one of the leading reasons why everything always seems to be more difficult than expected. In this post, I break down what fragmentation is, why it happens, and how to avoid or fix it. I very much look forward to comments regarding additional approaches — and will update this post accordingly.

What is data fragmentation?

One of the trends that has emerged across almost all industries and professions is to become increasingly data-driven. Sometimes this data is an integral record of the core business (e.g. keeping track of trading positions) and sometimes it is a record of behaviours (e.g. how many phone calls are made before a sale is completed), but either way: data is king.

But herding data is like herding cats — it almost seems like it is constantly doing everything it can to leak across multiple systems and databases! Before we know it, we have business critical information all over the place — with endless copies being made to try and bring things together. The speed of business slows down, the operational costs go up and regulators get unhappy.

Causes of fragmentation: Spreadsheets

The band-aid of the Fragmented Universe is the spreadsheet. As data demands increase, they multiply like rabbits until they’re the glue holding the whole business together. But spreadsheets are fickle beasts — with little to no audit capabilities and a poor multi-user experience. What starts as a band-aid evolves into a tight band restricting growth.

Causes of fragmentation: Multiple Parties

Sometimes, there are necessarily multiple individuals or companies with their own view on a particular data set — for example investors will receive a higher level view of a fund’s performance than the portfolio manager, and a fund admin will keep its own records. Other times, there are internal inefficiencies or limitations which force a company to split and copy its data between multiple places — for example between different asset classes or between multiple functions.

Causes of fragmentation: The Curse of the Quick Fix

Your software doesn’t do what you need. You talk to your vendor or your in-house team, and they tell you that they can’t fix it. You desperately wrack your brains for a solution and realise you can pull your trades out into Excel and apply your business process there. A year later an intern automates the process with some whizzy macro. The intern leaves and no-one knows how the process works, but hey — it works. Sounds familiar? This is often how system fragmentation starts…

Causes of fragmentation: The Difficulty Pyramid

System architectures are made up of three layers:

● Data — this layer defines how the data is represented and where it is stored. This is where all the records and information about a business is collected and persisted — commonly referred to as a ‘database’.

● Processing — this layer has all the business logic in it. It takes data from one or more sources and applies rules and manipulations depending on the needs and problems of the business.

● User Interface (UI) — this layer helps the user actually visualise the processed data. It also allows the user to capture information and configure what they want to see.

Each layer builds on top of the one underneath it — and just like a physical pyramid, switching out each layer gets harder as you go down due to the impact on the layer above.

The data layer is most important, but the hardest to change. Once years of data has accumulated within a system, there is huge fear around making changes. Decisions made years ago — possibly due to technical limitations at the time — become solidified as guiding principles. A classic example of this is simple American characters versus localised language support. If the data layer was originally set up to only support American characters, you can pretty much guarantee that adding multi-language support is going to be a gargantuan task.

Due to this difficulty, technology divisions under a lot of pressure cheat: they leave the data where it is, and put wrappers around it to make the user thinkit’s all from a single source. Sneaky. But then it falls apart, because some underlying limitation bubbles up and impacts the experience — “What do you mean the application doesn’t support Chinese characters — it’s brand new!”.

It is because of this pyramid that ancient code from the 70s still lives on in the global core financial system. The only solution is to tackle the problem head on and renew the technologies storing the underlying data.

It is worth noting that not everyone agrees with this assertion, and an increasingly common message is that we shouldn’t care how data is collected and stored — the only thing we should care about is how it is interpreted and visualised. This argument is primarily made with respect to Artificial Intelligence. I eagerly await the day when this is possible — but we’re a few years away from that yet!

Surely there’s a better way?!

It’s not all bad news! The technological advances of recent years have opened up new possibilities that help address the issues of fragmentation.

The most important technology to consider is the Application Programming Interface (API). An API is an interface that lets two or more computer systems talk to one another. APIs have existed since the beginning of computing, but it is only in recent years that they have become sufficiently standardised — enabling broad categories of applications to become integrated.

Why are APIs important for data fragmentation? Because they allow us to define a single ‘Golden Source’ of data, then integrate that with multiple applications. That single Golden Source then becomes the definitive, single copy of the data — in the past, the data would have had to be copied into every single application that needed it!

Data migration and consolidation

Where data of the same business type can be brought together into a single Golden Source that retains acceptable performance characteristics, it almost always makes sense to do so. Be ruthless in looking for opportunities to consolidate.

Do bear in mind that consolidating data of differing business types is sometimes possible, but not a good idea. A well architected platform should decompose differing data types into independent services, which can then be stitched together.

Conclusion

In conclusion, always remember that data is your most important asset — and that keeping it clean and non-duplicated will provide huge returns in the future.

If you have any comments or disagree with anything, please let me know!