Data Silos

Removing Silos in Planning - Part 3

Data silos, like process silos, are a type of functional silo that has been known and tackled by businesses for a long time. Like the latter, I will not rehash what has been more than adequately studied and published over the last decades. This will be the last part where I will only cover some high-level essentials from a technical angle to clarify the distinction with the other silo types. From the next part onward we will enter unchartered territory.

Data silos occur when the various business processes do not consume the same data whilst covering the same areas. Adjacent processes will have an overlap in data needed and higher level processes will need some aggregation of data used in lower level processes. Disconnects and misalignments of data between processes lead to myriad of problems, with many decisions not based on any reality. This leads a lot of companies to strive for a so-called "single version of the truth". This is data silo removal by a more catchy name.

Generally, when we need to align data, we can distinguish between 5 different types:

• Master data - this includes codes, names, descriptions, many attributes, and relations between mostly static data. Examples are product, locations, customers, bills of materials, routings, and so forth.
• Transaction data - this includes sales orders, purchase orders, shipment orders, production orders, and so forth. Generally, it is things that have executed, are in the process of being executed, or execution is imminent.
• Planning data - this includes forecasts and plans of all types. Unlike transaction data, planning data is transient in nature. Each time we plan or forecast we can override the previous iteration.
• Analytics data - this includes any data used for analytics, accounting, reporting, and so forth. It is different from the prior three in that it is value data rather than identifiable data (it does not need a record ID) and is more transient even than planning data.
• Local data - this is data that has no real need to share or align on. This is often system-specific data required to make it function properly.

When companies start removing data silos the first step is to normalize master data. Master data is like the lingua franca of enterprise data. You cannot share any other data without a common frame of reference. The bigger the company gets the harder master data normalization becomes. Some, having grown through acquisition or merger, may have hundreds of ERP, CRM, MES, LIMS, and SCM system instances. They may have great overlap in master data, but it is all coded differently. If one system has version X of product A, is it the same as version 2 of product 3 in the other system? Different systems will often have different number of keys. Where in one system a record may be identified by code and version, in the other it may be name, variant, and effective date. Massive mapping tables that will need to be perpetually maintained are the result. As more systems are involved complexity grows exponentially.

For the most part, companies, through enormous investment of time and resources, have been able to normalize their master data to significant degree. An exercise they keep repeating every time a new acquisition is made. Often this data will now sit in some central data warehouse and be accessible to other systems.

Once the master data is under control, connecting the dots on transaction data is a much simpler task. Typically, transactions will contain the ID of the other party already. If two divisions sell product to each other, an inter-company purchase order for one will match a sales order for the other and the PO number will be known for the sales order. Even if both used different master data, they can be matched up. Some companies have also automated the reverse. Using AI to match master data across systems by virtue of being able to connect some of the transactional records on both sides. Naturally, exceptions are plentiful, so manual labor will still be substantial.

Planning data presents a different kind of problem. Since it is transient (short-lived), for the most part historical planning data does not need to be normalized. It just requires effort to design and build the means that ensure future planning data is normalized. This means exceptions that need human attention can be handled whilst the memory is still fresh, making it less error prone. Historical data is often forgotten, or biased, or even occurred when other people performed the role. Was this huge order 3 years ago a data entry error, or did it really occur? For transaction data that is important but hard to ascertain. For planning data we only care if it is recent or upcoming.

Planning data does introduce a whole new bucket of worms which will be covered in the next parts in this series.

Analytics data is generally reproducible as long as the underlying details (of the other 3 types) are aligned. There are some exceptions, in which case the problem is similar to dealing with planning data.

All of the above basically leads to a state where all systems can now communicate through translation. Many companies will take the next step to actually replace data inside systems or more often even replace entire systems to make data match across them. This leads to a much more maintainable situation from the data perspective.

However, it typically leads to deterioration from a process perspective. For example, when a company standardizes on their ERP system, some parts of the company that had a best-of-breed ERP for their unique requirements will find the new system forced upon them cannot solve all their critical business problems. The use of spreadsheets to bypass the limitations of the new system generally explodes. The image from the last part on process silos (repeated here), contains both process and data silos.

As data silos get removed through standardization, many of the databases (shown as drums) merge into fewer larger ones. But the number of spreadsheets (shown as Excel icons) multiply, and processes become more fragmented again (introducing new process silos).

To avoid this conundrum a whole new type of system has emerged onto the market. They offer Low-Code Application Development (LCAD) and the systems are named Low-Code Development Platforms (LCDP). They allow rapid development of simple custom applications allowing companies to fill gaps in functional needs between the large enterprise systems. They thus serve the exact purpose spreadsheets have historically served, but without the ensuing versioning and process hell. They are managed by corporate IT, follow all IT requirements for security, backup, and recovery, but can be developed by business users anywhere in the organization without any technical expertise. Additionally, they can serve as a Master Data Management (MDM) bridge, filling master data holes where critical systems are missing places to store such data themselves. This applies especially to master data needed by the custom applications themselves.

In this new hype, many such platforms are now offered commercially. The market leader for general-purpose LCAD is Mendix, whilst in planning-heavy environments more purpose-built alternatives like Flowfactory make it easier to develop gap-filling solutions with lower required skillsets. I would highly recommend any company that is still struggling with spreadsheet mayhem to explore LCAD.

It is my assessment that the problem of data silos is generally well-known, understood, and tackled to a large degree. But work is still left to do. It always is. Starting with the next part, I'll cover silos that have not had nearly as much attention. Undeservedly so.

This article is part of a series covering the 4 types of functional silos in planning:

• Part 1 - Introduction
• Part 2 - Process Silos
• Part 3 - Data Silos (this article)
• Part 4 - What are Information Silos
• Part 5 - What are Decision Silos
• Part 6 - How To Integrate Information
• Part 7 - A Decision Framework Blueprint (link to be added once published)
• Part 8 - The Data-Driven Enterprise (link to be added once published)

Find all my articles by category here. Also listing outstanding articles by other authors.