How do you manage data silos, duplication, inconsistency, and incompleteness?

Threats of data silos: - Arsises risk for data integrity. - Resource wastage. - Discourages collaborative approach. Here's how to avoid data silos: - Filter and consolidate data to centralized space. - Store data that is relevant to your goals. - Keep updating stored data.

With my +20 experience in multilateral organizations, I have seen this disintegrated approach, across all vertices. Not that it is designed, but because it's never thought through that way. Even within single organisation, multiple platforms performing overlapping functions have different platform of data.

Data silos can impact the insights generated from data analysis and also create differences in understanding of business objectives between different departments.

The average organization deal with a number of different enterprise applications to manage their business (CRMs, ERPs, etc…). It’s very hard to find one solution to manage everything and as the organization complexity grows, find the one size fit all solution became impracticable. This naturally lead to the isolated data silos. Those application inevitably have to manage the same informations (customers, employees, invoices, orders, …) and since they are disconnected from each other, over time the different way to update, manage, and deal with it lead to inconsistency and redundancy.

Let's talk about data silos. Similar to "silo working", this causes data isolation and can make it more difficult to see patterns and insights. For example, having transactional/sales data in one place, and customer engagement data in another makes it a more difficult journey to identify strengths, weaknesses, opportunities and threats in your customer sale funnel. Constructing a data warehouse with strategically built data pipelines can allow you to bring data into one consolidated place, allowing you to easily analyse business data patterns and produce insight to drive forward your business. As cloud services have become cheaper, these solutions are now easily teachable for SMEs with the right help from a consultant or agency.

Data inconsistency and duplication can come from entering the same data in different source systems without data governance. Having data governance/data dictionary can prevent having to perform major data cleanups when trying to migrate into one main source system and/or integrate with other systems within the same organization.

This issue is prevalent in organizations that lack a data-driven approach. Consequently, various departments establish their own miniature IT, data engineering, and data science teams, each independently handling data within their respective domains. The article correctly highlights that this practice leads to problems such as redundant data, a lack of agreement on results, and challenges in making informed decisions.

Duplicated data: Datasets that are duplicated in multiple locations. This can cause confusion as there is no "single source of truth" for the dataset. Inconsistencies: These are often present in duplicated data sources and can contribute to confusion. One data source may say one thing and another entirely different, common causes of this are viewing duplicated data across variable timelines. Data incompleteness: Missing data and lost data can also result from duplicated data sources or a lack of data quality. All of the above can be addressed and mirigated through the right strategy and goal of consolidating data into one central point such as a data warehouse.

Data duplication & inconsistencies manifest when data users attempt "convenient" data corrections, such as resorting to creating new accounts due to system limitations that restrict modifications to account usernames post-creation. In principle, data duplication prevention should be universally upheld, but practical constraints often render some duplication & inconsistencies inevitable. A pragmatic approach involves systematically quantifying the extent of duplications, inconsistencies, and incomplete data within the system. Subsequently, corrective actions should be taken at the source over time, with ongoing measurement of the effectiveness of these efforts.

Another issue that can affect data reliability is it’s currency. Data update frequency requirements need to be established based on the type of data, how often it changes and how is utilised in the decision making process.

Human error during data entry is a common cause of data challenges. Typos, misinterpretation of data, and mistakes during manual input can introduce inaccuracies and this can be eradicated through training and educating personnel on accurate data entry techniques, implementing validation checks, and using data entry software with error-checking capabilities can help reduce these errors.

Often a lack of knowledge in the business around data and digital transformation contributes to problems like these. As businesses go through digital transformation it is important to conduct a full audit the data within the business and sense check/create a strategy that ensures data becomes as insightful, valuable and actionable as it can be within the business. When businesses skip this step, digital change is often implemented in disjointed ways in an organisation, leading to multiple data sources and systems, and creating the difficulties mentioned in this article.

This problem primarily arises from the lack of executive dedication to instituting a data democratization effort. Realizing true potential requires aligning all layers of data in harmony.

These data challenges arise due to a lack of adequate management support for treating data as an asset. Consequently, data consumers, including data scientists, resort to inventing solutions when utilizing the data. This leads to a situation where data is not effectively managed at its source, resulting in data consumers operating in isolated silos, thereby contributing to these data challenges.

In my experience these challenges are due to A) Lack of knowledge on data needs - used for Reporting, Analytics or BI etc.., B) No standardised data sourcing systems - Garbage in - Garbage Out C) Missing Data dictionaries - No clarity on data definitions D) Databases not built to scale - Tables start to disintegrate due to size

Securing corporate buy-in is imperative. To address this challenge, one effective approach is to establish a centralized data hub within the organization. This involves creating a corporate data lake with data pipelines directed to this central repository. Implementing a robust governance program, complete with well-documented data definitions, controlled data access, and robust security measures, is vital. Furthermore, departments should have the flexibility to leverage various data layers and employ their preferred analytical platforms for conducting analyses. The key to achieving data democracy lies not in imposing restrictions but in granting the freedom to innovate.

Implementing Data Governance within the organization encompasses a range of activities. These include, but not limited to, identifying the organization's most critical data, assessing its intrinsic value as an asset, setting up a master data management, designating data owners, creating matrices to gauge system-of-record adoption, and assessing data quality by implementing data quality rules. It's important to note that this is not a task for a single individual but rather necessitates robust management support to establish the framework, active accountability from the business stakeholders, and continual end-user training to enhance data literacy.

Audit and strategy are the foundations and essential building blocks needed to overcome challenges faced with inconsistent, incomplete and disjointed data sources. Creating a well-versed strategy that involves all stakeholders and data handlers from the start ensures that everyone is on board. This can make sure that the right infrastructure can be put into place, such as data warehousing, to enable data to be used as a powerful enabler to decision making in the business .

In Revenue Operations, managing data challenges is pivotal for success. To address data silos, duplications, inconsistency, and competence issues, RevOps experts focus on aligning teams, optimizing data management, and leveraging technology. They break down silos through integrated data management tools, ensure data consistency via governance policies, and cultivate a culture of data competence among team members. This harmonization of people, processes, and technology enables businesses to make informed, agile decisions, ultimately maximizing revenue potential. In the dynamic landscape of RevOps, adaptability and continuous refinement are key to achieving revenue excellence. Did you hire your revops expert yet?

Implement validation rules and checks at the point of data entry to prevent errors and inconsistencies. This includes checks for data types, ranges, and dependencies but also use data profiling tools to analyze data quality issues and identify anomalies, outliers, and patterns that need attention.

Having a data dictionary is vital. A data dictionary of your source system will give the data location, field name, what that field is used for, an example of the data format and an example of what the data should look like when entered into to system. Anything outside of what is in the dictionary is not allowed or need to be discussed internally as a whole to determine it’s value.

One of the techniques and resources for efficient data management is data security. It has rules and processes that are intended to safeguard data from unauthorized access, security flaws, and online dangers. For a number of reasons, it is necessary to execute strong data security practices, including Protecting Confidentiality, Preserving Integrity, Ensuring Availability, and Mitigating Financial Risks.

The interpretation of Data Management can vary among individuals. I recommend referencing the Data Management Body of Knowledge, which provides a comprehensive framework encompassing data management's best practices, principles, activities, and tools. While data catalog and lineage information may not require advanced expertise, in some instances, they can be maintained through simple lists or Excel files. Basic SQL or Python scripts can be employed for measuring and managing data quality. However, when dealing with enterprise-level data management, it becomes imperative to oversee these processes in a controlled manner. Collibra is a dedicated Data Management platform that I have utilized in the past in an enterprise setting.

Establish data backup and recovery procedures to safeguard against data loss due to hardware failures, cyberattacks, or other disasters

Tips for data management: 1. Establish a clear data ownership procedure, which department are responsible for what data. 2. Standardize your format across company, whether for dates, weight, numbering, hierarchy. 3. For large entity, consider having a master data / cataloger department.

Data can be in silo. But its time to bring it in front of everyone to maintain consistency. To give access to data: Its should be properly organized Access to data based on critically/user role is highly recommended. Ownership is also very important to keep it updated, curated, managed.

Steps to consider - 1. Framework: Develop governance 2. Inventory: Centralized, cataloging with metadata 3. Integration/ETL: For data connection, transformation 4. MDM: Implement Master Data Management for consistency 5. Data Quality Assessment: Regularly 6. Cleansing/Enrichment: Correct errors, enrich 7. Documentation/Lineage: Maintain data lineage 8. Data Access Control: Enforce, for security 9. Quality Monitoring: Set up real-time monitoring 10. Training/Awareness: Educate on best practices 11. Data Stewardship: Appoint domain-specific stewards 12. Retention/Archiving: Set policies 13. Auditing: Perform for compliance 14. Continuous Improvement: Enhance processes continually 15. Governance Council: Form a cross-functional body