Breaking Down Data Silos

One of the biggest challenges your data science team is likely to encounter is gaining access to all of the organization’s data. Many organizations have?data silos— data repositories managed by different departments and isolated from the rest of the organization.

The term “silo” is borrowed from agriculture. Farmers typically store grain in tall, hollow towers called silos, each of which is an independent structure. Silos typically protect the grain from the weather and isolate different stores of grain, so if one store is contaminated by pests or disease all of the grain isn’t lost. Data silos are similar in that each department’s database is separate; data from one department isn’t mixed with data from another.

Data silos develop for various reasons. Often they result from common practice — for example, human resources (HR) creates its own database, because it can’t imagine anyone else in the organization needing its data or because it needs to ensure that sensitive employee data is secure. Data silos may also arise due to office politics — one team doesn’t want to share its data with another team that it perceives to be a threat to its position in the organization.

If your data science team encounters a data silo, it needs to find a way to access that data. Gaining access to data is one of the primary responsibilities of the project manager on the data science team. After the data analyst identifies the data sets necessary for the team to do its job, the project manager needs to figure out how to gain access to that data.

The Problems with Data Silos

Although data silos may be useful for protecting sensitive data from malware and from unauthorized access, they also cause a number of problems, including the following:

• With data silos, an organization has no single source of truth. Data from various departments must be collected and combined prior to analysis.
• If two or more departments are storing the same data, figuring out which department has the most accurate and current data can be a major challenge.
• The chance of overwriting new data with old data is increased.
• Data sharing may be more difficult and less efficient.
• Data security may be more challenging if the organization needs to secure multiple sources of data, as opposed to having data in only one location to secure.

I once worked for an organization that was trying to migrate all its data to a central data warehouse. They felt that the organization wasn’t getting enough insight from its data. The organization had just gone through a data governance transformation and wanted to govern how the data was controlled in the organization.

When they finally got into their data, they realized how much was locked away in silos that no one knew about. Over many years, each department had created its own processes, schemas, and security procedures. The organization wanted to get value from this data, but that data was stored on different servers across the entire company. To compound the problem, the various departments were reluctant to share their data. It was as if the project manager was asking them to share toothbrushes.

Breaking Down Data Silos

One of the first steps toward becoming a data-driven organization is to break down the data silos:

1. Migrate all of the organization’s data to a secure data warehouse. A cloud data warehouse may be the most economical, because you can outsource data warehouse management and security to a third-party vendor that has the technology and expertise to provide superior performance and security.
2. Assign each user a unique username, and require a secure password from each user to log in. This enables IT to grant unfettered access to all data they may need to do their jobs, while restricting unauthorized access to any sensitive data.
3. Provide users with the tools and training they need to query and analyze the data.

By breaking down data silos, you give everyone in your organization self-serve access to the data they need to do their jobs better.

Words of Advice for Project Managers

If you’re a project manager on a data science team, try to keep the following key points in mind:

• Don’t underestimate the difficulty of gaining access to data stored in silos. It may take a long time, so get started before the team actually needs the data.
• Migrating an organization’s data to a centralized data warehouse requires the entire organization to be on board. You’ll need executive buy-in to make any progress. You can also expect to have to sell each department on the idea. Expect push-back from some departments that are highly protective of their data or that think “if it ain’t broke, why fix it?” You may have to entice them by explaining that with central data storage they’ll be able to create more complex reports or use newer visualization tools.
• Provide access to your team’s reports. You may have an easier time breaking down silos if you can show the value of company-wide reporting and insights. Build interest in your information system by sharing your team’s wins. When others in the company see the value in the data and the BI, they’ll be eager to adopt.
• Do your best to protect the data science team from any meetings about breaking down the data silos. You want the rest of your team focusing on exploration and discovery, while you focus on getting them access to the data.

Frequently Asked Questions

Why do Data Silos occur?

Data silos occur when data is stored in different systems or departments and is not easily accessible across an organization. This can result from a lack of data integration, different data management systems, or organizational boundaries.

How can I identify data silos within my organization?

To identify data silos, look for departments or teams that have their own separate data management systems and are reluctant to share data. Also, inconsistent data and information silos can be indicators. Regular audits and reviews of your data practices can help you spot these silos.

Why is breaking down data silos important for data management?

Breaking down data silos is crucial because they prevent the flow of information across an organization, leading to inefficiencies, duplicated efforts, and missed opportunities. Effective data management requires a comprehensive view of all relevant data to make informed decisions.

What are some common ways to break down data silos?

Common ways to break down data silos include implementing a unified data platform, centralize your data storage in a data lake, encouraging data sharing across departments, enhancing data governance practices, and using data integration tools to combine siloed data.

How can data governance help in eliminating data silos?

Data governance helps in eliminating data silos by establishing policies, standards, and practices for how data is collected, managed, and shared. It ensures that data across the organization is consistent, accurate, and accessible, which reduces the chances of information silos forming.

What is a data lake and how does it help in breaking down data silos?

A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. By centralizing your data in a data lake, you can break down silos, making it easier to integrate and analyze data from various sources.

Can a data mesh approach help in eliminating data silos?

Yes, a data mesh approach can help in eliminating data silos by decentralizing data architecture and allowing different teams to own their data endpoints. This encourages data sharing and integration across the organization, reducing the barriers that create data silos.

How does data integration help in managing siloed data?

Data integration involves combining data from different sources to provide a unified view. This helps manage siloed data by breaking down barriers between different data sets, allowing for better data analysis, improved data quality, and more informed decision-making.

What role does technology play in eliminating data silos?

Technology plays a crucial role in eliminating data silos by providing tools and platforms for data integration, data governance, and data sharing. Solutions like data lakes, data mesh, and unified data platforms help centralize and manage data more effectively, breaking down the barriers created by data silos.

How can breaking down data silos improve customer experience?

Breaking down data silos can significantly improve customer experience by providing a single view of customer data across all touchpoints. This ensures that customer interactions are consistent and personalized, leading to higher satisfaction and loyalty.

This is my weekly newsletter that I call The Deep End because I want to go deeper than results you’ll see from searches or AI, incorporating insights from the history of data and data science. Each week I’ll go deep to explain a topic that’s relevant to people who work with technology. I’ll be posting about artificial intelligence, data science, and data ethics.?

This newsletter is 100% human written ?? (* aside from a quick run through grammar and spell check).

More sources

1. https://www2.deloitte.com/mt/en/pages/strategy-operations/articles/mt-breaking-down-data-silos.html
2. https://pipeline.zoominfo.com/operations/how-to-break-down-data-silos
3. https://blog.hubspot.com/service/data-silos
4. https://hbr.org/2016/12/breaking-down-data-silos
5. https://business.adobe.com/blog/basics/what-are-data-silos-and-how-can-you-eliminate-them
6. https://datavid.com/blog/eliminate-data-silos
7. https://www.talend.com/resources/what-are-data-silos/
8. https://softteco.com/blog/what-are-data-silos-and-how-to-eliminate-them
9. https://www.starburst.io/data-glossary/data-silos/
10. https://www.secoda.co/learn/break-down-data-silos-to-fuel-growth
11. https://www.sitecore.com/resources/customer-data-management/eliminate-data-silos