How can you manage data quality when integrating and transforming banking data?

Data Quality can mean different things to different personas/stakeholders hence might lead to ambiguous requirement. I find it incredibly important to streamline the DQ purpose(Data engineering/Data Analytics/Data Science/Reporting) and then connect back with the DQ requirements and refine them. This would help in creating a purpose driven DQ solution.

Data quality is very important and various organizations realize it when they see use cases spending lot of time in chasing and fixing data quality issues with source or platform. Shying away from doing initial investment in establishing data stewardship(data quality as one of the core responsibilities) causes lot of pain in future. With so many different types of data stores today the quality checks become even more important. Data quality should be checked at the sources and aggregator level so that all downstream use cases(Analytics, Risk/Fraud, Marketing, Regulatory, ML/AI) get consistent quality data fit for purpose. Depending on the environment data quality can be implemented using custom build, cloud native and manger services.

Banking Data is required a maintain minimum level of accuracy, conformity, consistency in the core systems and hence its extremely important to look at DQ in those systems before starting to integrate. Identifying system of records and conducting a system level certification is critical. Fixing the data at source is critical part of this and should be budgeted in any integration project.

A strong business use case and ROI model is the foundation to implement AI in trading world. This is to ensure the transaction cost per trade does not increase.

Cross match both data, before the basic fundamentals of accounting the data accuracy and anlisizing and scrutiny is mandatory.

There are many AI driven data observability tools available now that can evaluate and correct data drifts. In addition to that masting systems also have the Ai driven DQ capabilities that should be evaluated.

Based on organization maturity I have seen data quality tools and services provided are either centralized or de-centralized. Both approaches have there pros and cons. Data profiling is integral part of data quality ensuring that ongoing data quality is also utilizing sophisticated ML techniques specifically for time series data. Lack of well established SLAs/SLOs between source and target is one of the mail paint points across the board. Data quality issues can be well integrated with incident management to ensure that data steward and technical support is informed proactive or detective way depending on key data parameters- sensitivity, SLA/SLO, producer, consumer, steward, incident queue, alerts.

In my experience,it is always recommended to resolve DQ issues arising from data entry at the source rather than handling during integration. Nearly 30% of DQ issues arise due to non standard data or incorrect data entered by users. For example, users not having an option to select from predefined values for some key demographic information like gender,race, ethnicity/language or no field level validation on zip code/email-id or no validation for missing address and so on. This is leading to lot of data inconsistency/duplicacy/missing values during reporting. This can be handled by eliminating free text boxes and implementing drop down/multi select check boxes or applying screen level validation on zipcode/emailid columns and so on.

One of the most important aspect of data quality is designing data quality rules. Once the data quality requirements are assessed,critical data elements (CDE) are identified and profiling is performed, one should start listing the possible DQ dimensions that are feasible for CDE (Not all dimensions are valid for a single CDE). Refer the business glossary for the CDE for listing the dimensions and drafting the rules. The taxonomies in the glossary can guide you in drafting these rules Walkthru these rules with the data stewards and business users for key inputs. This might take multiple iterations before the rules are finalized and implemented. At CommonSpirit Health , we use power query to simulate DQ rules before implementing them.

A DQ dashboard is a potential solution to test and track the DQ scores. This can be as simple as counting NULLS or as complex as performing customer analytics. Plethora of DQ/Data intelligence SaaS solutions available that can let us track DQ health.

Monitoring and improving data quality (DQ) using AI-based tools is a strategic approach for maintaining accurate and reliable data within your organization Design automated workflows that initiate data quality improvements based on predefined rules and thresholds. Implement feedback mechanisms that enable data stewards and users to report and address data quality issues. This feedback loop is essential for ongoing improvements. Ensure that your data governance practices are in sync with your AI-based DQ initiatives. Clearly define roles and responsibilities for maintaining data quality. Document your processes and provide training to your team on effectively utilising AI tools to maintain data quality

For any data quality program to be established there are certain pre-requisite like -: 1- Well Established data stewardship/custodianship/ownership 2- Data quality has understanding and sponsorship at enterprise level using top down approach. 3- Definition of data quality is well established - Controls vs quality checks We should look at some of the latest frameworks like CDMC which provide some details on data quality as one of its key pillars. In addition of data quality definition and key features needed, we should look at some user experience and non functional(performance, access management, integration with other services - governance, incident management) features as well as we build or buy the tools/services.