How do you confirm your data architecture is sound?

Data quality includes system audits to assess the handling of personally identifiable information or PII. Proper representation of PII requires adherence to standards and laws such as GDPR and HIPAA.

Data quality also includes how the system processes the data and how and where calculations are done. In an ideal world the same calculation would yield the same result no matter where it's done in the process. In the real world this isn't the case, usually because of the way data gets processed till it reaches an end-user. The architecture needs to provide a reference for these anomalies and guidance as to what the correct or preferred outcome.

Steps involved in our data architecture journey - Clear Objectives, Data Profiling, Data Modeling, Scalability, Data Security, Data Integration, Performance Testing, Documentation, Monitoring and Maintenance, Feedback Loop.

To have a sound Data Architecture, it requires a thorough assessment of various aspects of data infrastructure, data sources, storage, processing, security, and scalability. Here are steps to help you evaluate the soundness of your data architecture: 1. Data Strategy and Governance 2. Data Sources 3. Data Storage 4. Data Integration 5. Data Modeling 6. Scalability 7. Performance 8. Data Security 9. Backup and Disaster Recovery 10. Documentation 11. Monitoring and Logging 12. Cost Efficiency 13. Data Quality 14. User Feedback 15. Compliance 16. Future-Proofing

This is a common issue as not enough controls are in place to qualify data in frontend systems. All data architecture needs to have robust data profiling and validation controls at design phase. Depending on an organisation a separate tool can perform these or data quality checks need to be part of every stage when moving data from source for Insights, machine learning or reporting purpose.

It's also important to keep on evaluating your data based on usefulness. Keeping in mind that the raw unprocessed data usually is stored in a type of cold/archive storage, is it necessary to have all the data you currently have readily accessible even if it's not being used? Keeping old and/or irrelevant data just for the sake of having everything available can hinder speed performance and potentially also inflate associated costs.

Data performance can be evaluated by following below steps : Start by defining clear goals: Determine what you want to achieve with your data architecture, like faster data processing or better data quality. Measure key metrics: Track important indicators like data processing speed, accuracy, and reliability regularly. Compare with benchmarks: Compare your data performance metrics with industry standards or past performance to identify areas for improvement. Identify bottlenecks: Pinpoint any areas where data processing is slow or errors occur, and address these issues. Adjust and iterate: Make necessary adjustments to your data architecture based on your findings and continue to monitor and improve over time.

Designing solid data models, executing effective data integration, guaranteeing data security and compliance, and optimising data storage and performance are all examples of quality contributions to data architecture. Effective documentation, business objective alignment, and cross-functional team communication are essential. It's important to stay up-to-date on new technology, problem-solving techniques, and to maintain open lines of communication with non-technical stakeholders. Delivering valuable data solutions that enable organisations to make educated decisions and accomplish their strategic goals requires ongoing improvement and project management expertise.

Performance considering is a must as end users as customer require better performance from newly designed solutions. Usually, organisations do not have enough performance metrics to create a benchmark. That said, architecture should ask for some benchmark metrics to validate some assumptions that may have been applied in the design. These metrics provides a clear expectation on the final delivery of the system.

Decision matrices and decision logs assist to surface data architecture patterns as well as user needs. Data observability tools and platforms are useful to monitor data quality and make the case for data optimization efforts.

Data catalogs, data dictionaries, data lineage, data stewardship, and data audits are invaluable tools and techniques for this purpose. They not only help in documenting and tracking data assets but also empower data stewards and teams to control and optimize the data lifecycle. By having a well-established data governance framework in place, you're not just complying with regulations and mitigating risks; you're unlocking the full potential of your data. It allows you to harness the insights, make informed decisions, and drive innovation. In essence, sound data governance is the linchpin that turns data into a strategic asset that propels your organization forward in today's data-driven landscape."

Data governance is just one part of an organisations governance framework and should not be looked in isolation. At times architecture is created for a particular business requirement and reviewing organisation governance framework will provide details on all governance aspects to be considered in the architecture. Following the organisations governance framework will provide sufficient artefacts to avoid any architecture flaw in ones design.

One thing I've found helpful is to test your data strategy to see if it delivers Findable, Accessible, Interoperable, and Reusable (FAIR) data. FAIR data is a critical need if any organisation wants to create enduring data assets which can be exploited and provide value beyond their original intent. The breakthrough question to ask is "do the processes, people, technology and content outlined in our data strategy ensure that we deliver data that is FAIR for the enterprise?" When you find some shortfall that's where to focus the gap analyses, maturity assessments and SWOTs

Domain modeling and analysis are methodologies used to assess gaps and strengths in data architecture. Understanding what is working and what needs attention can focus a data strategy by highlighting deficiencies in a system.

This aspect, in my opinion is the most important way to measure effectiveness. Most of the other steps are basically the hygiene factors you find in DAMA-DMBOK. These are important basic building blocks and need to ge in place, but they do not measure or optimize effectiveness or soundness. You need to measure and optimize. Align to needs and business strategies.

Feedback channels are crucial to supporting an extensible and useful data system. Ultimately, end users, stakeholders and regulators are your customers and consumers of data.

Even before one starts on a new data architecture design, enough consultation has to take between the end-users and their expectations to guarantee success or the final design. Having a end-user as stake holder in final review and feedback is effective.

- The perspectives of Data-Analysts and Business-stakeholders is very much the way to judge the success of Data Architecture. - In my view it seems not so easy to bring the Cost parameter into consideration in NEW spaces. In organisations that have produced insights for a few years it would be easier to bring-in cost. - Cannot over-emphasise the need to persist with a Technology Architecture that is producing results, in order to tune it towards higher ROI.

Be lean and agile up to six-sigma levels: identify the current bottleneck, solve it to scale 10x, re-examine the architecture, identify the next bottleneck. Also, pay special attention to cost: ensure it grows linearly with volume; steep or even geometric growth of cost usually is usually due to an architectural issue. Retest the whole architecture with each iteration: some issues may reoccur (known in IT as regression).

"culture eats strategy for breakfast"! It's becoming a cliche, but it is a huge influence in how well data lands in an organisation. Enterprises that are most successful at getting enduring value from their data are those who consider interoperability from the start. We need to develop a culture where; the act of data creation is intimately coupled with the act of information sharing, and individuals and groups continue to produce and consume information in ways that are speci?c to their own business needs, but they produce it in a way that can be connected to other aspects of the enterprise.

Stress test before committing: start with a PoC, and make sure the final product would be able to support the velocity, volume, veracity and variety needed. If you can't ensure a 10x growth of each without issues, you need to get back to the drawing board. Also, keep an eye on cost at all times and project for 10x and 100x volume increase. That way, one can always tell (hopefully in advance) if the cost becomes restrictive and adaptations to the architecture are required. Lastly, ensure cost grows linearly with volume; steep or even geometric growth of cost usually is usually due to an architectural deficit. Time to get back to the drawing board!

Sound data architecture is a cornerstone of data-driven decision-making and innovation in organizations. Regular assessment, alignment with business goals, data quality management, scalability, security, and ongoing communication are key elements in ensuring the effectiveness and relevance of your data architecture. Periodic reviews and adjustments will help keep your data architecture in sync with your organization's evolving needs and the dynamic data landscape.

it's crucial that data architecture aligns with your business goals and meets your data processing and storage needs efficiently. Start by assessing whether it can effectively handle the volume, velocity, variety, and veracity of your data. Check that data sources are well-integrated, data flows smoothly through the architecture, and data quality is maintained. Evaluate the scalability, security, and compliance aspects of your architecture, and confirm that it supports your analytics and reporting requirements. Regular monitoring and performance testing can help identify and address any issues, ensuring that your data architecture remains robust and reliable over time.