Your data architecture is riddled with discrepancies. What steps should you take to uncover the root cause?

In a previous project, we encountered a significant data discrepancy during a migration between legacy and cloud-based systems. Initially, data seemed to transfer smoothly, but upon closer inspection, we discovered that certain fields were not mapping correctly due to outdated schema definitions. This led to inconsistencies in reporting and analysis downstream. By conducting a thorough review of our data mapping and transformation processes, we implemented stricter validation checks and improved documentation, ensuring smoother data migrations in future projects.

When data discrepancies arise, I meticulously assess the data flow to uncover the root cause. I trace data from its origin, through each transformation and integration point, to its final destination. By examining the entire journey, I can pinpoint where errors or inconsistencies might be introduced. This often involves analyzing data pipelines, transformation scripts, and system logs to identify potential culprits like faulty data sources, incorrect mappings, or coding errors.

Being a leading data architect, my approach is to ensure that data design is an integral component of enterprise data feeds. Irrespective of the sourcing of data, whether real-time, batched, files-based, live external connections - we need to ensure that data design is first step in onboarding data. This approach works by having data model and dictionary(s) being current. Anytime, there is requirement for new or modified data, analyze the needs and update your data models and architecture appropriately. Data flows are critical to ensure data availability and to prevent stale data. Data redundancy, data formats, transformations, and external identifier mappings could result in inconsistent data. [Continues in next section]

When facing data discrepancies, reviewing established standards is crucial. I revisit our data definitions, naming conventions, and validation rules. Are they up-to-date and consistently applied? By comparing the actual data against these standards, we can pinpoint deviations and identify potential sources of inconsistencies. This thorough review helps us uncover the root cause of discrepancies, whether it's outdated standards, inconsistent application, or a misunderstanding of the guidelines.

As we saw in previous section, data design is an important element for ensuring correct handling of data onboarding and continued management. There is another benefit of data design - governance. Data design should indicate sensitive, confidential, or controlled elements. Data governance team can use this information to ensure correct handling of data according to regulations and governance requirements. Design with data ownership, source of truth, sensitivity levels, encryption needs, and compartmentalization requirements will allow an organization to evolve data and grant access to data consumers per policy. Lack of ownership and standards are usual reasons for data discrepancies in enterprise data. [Continues in next section]

Data discrepancies often arise during integration processes. To uncover the root cause, I carefully examine how data is being transferred and transformed between different systems. I look for issues like incorrect mappings, data loss during transfer, or errors in transformation scripts. By scrutinizing the integration points, I can identify where inconsistencies are introduced and take steps to fix them, ensuring that data flows smoothly and accurately throughout the entire architecture.

Data integration is not just limited to APIs in a literal context. Rather, this means how you assimilate data elements from several sources to create unified and harmonized data sets. As we saw in previous section, data unification is dependent on identity elements and resolving duplicates. Ensure that your data team is paying attention to transformational errors or rejected data and take corrective measures. Otherwise, even generative AI cannot solve the GIGO [garbage in, garbage out] state of your enterprise data. [Continues in next section]

Anyone who is a participant in your entire data journey, from sourcing to consuming is a stakeholder. However, different stakeholders necessitate different communication and engagement methods. For example, your data transformation team(s) need both technical and business rules to ensure proper transformation of source data. On the other hand, business users are only interested in availability of harmonized data to provide full 360° view of customer's data. Assurance to downstream data consumers for data quality is a paramount concern for data administrators. Your intra-team and inter-team communications must use precise details for appropriate data detangling and quality resolution.

Establishing data design, model, dictionaries, and standards will allow you to setup autonomous monitoring and alerting mechanisms. Define metrics on data resolution and transformation issues, and institute periodic architectural reviews to provide correctional guidance for resolving issues strategically.

Begin by conducting a thorough audit of the data sources, processes, and transformations involved in the architecture. Review data lineage to trace where and how data moves through the system, identifying any points of divergence or inconsistency. Utilise data profiling and quality assessment tools to analyse data integrity issues, such as missing values, duplicates, or anomalies. Collaborate closely with stakeholders across different teams, including data engineers, analysts, and business users, to gather insights and perspectives on potential causes of discrepancies. Implement rigorous testing and validation procedures to verify data accuracy and consistency across various stages of the architecture.

Data Audit: Conduct a thorough review. Source Analysis: Examine all data sources. Consistency Check: Verify data formats. Data Mapping: Compare source to destination. ETL Review: Analyze transformation processes. Validation Rules: Check data validation criteria. Error Logs: Investigate logged errors. Stakeholder Input: Gather feedback from users. System Updates: Ensure systems are current. Documentation: Review data documentation. Test Cases: Create scenarios to identify issues. Collaborate: Work with team for insights.