Exploring Data Quality: Insights from 'Data Quality Engineering in Financial Services' Book

Recently, I've been immersed in the book "Data Quality Engineering in Financial Services" by Brian Buzzelli making my way through the first couple of chapters. It's been an enlightening journey so far, especially delving into the concept of data as a valuable asset for businesses.

One of the ideas introduced in the book is the concept of Data Quality Specifications (DQS). Essentially, it's about collaborating with process/function experts to define what makes data acceptable both when it's extracted/generated and when it's used by people or other systems.

To achieve this, the book suggests using a framework called "Data Dimensions" to gauge the quality of our data based on certain measures.

In simpler terms, it's about checking our datasets against specific criteria:

• Completeness: Are all the required data fields there?
• Timeliness: Is there a timestamp associated with the data?
• Accuracy: Is the data correct? Defined by someone who knows the business or function well.
• Precision: How detailed and accurate are the numbers?
• Conformity: Does the data meet any standards?
• Congruence: Do the data from different time periods look similar?
• Cohesion: Are the values consistent across different sets of data?

By setting these Data Quality Specifications, we're equipped to evaluate how well different datasets measure up.

Here's an example illustrating how we might assess these dimensions:

• Completeness: All customer records include name, address, and contact information.
• Timeliness: Sales transactions are logged with a timestamp within one hour of occurrence.
• Accuracy: Customer account balances match actual financial statements.
• Precision: Stock prices are recorded to two decimal places.
• Conformity: Data follows GDPR regulations regarding personal information.
• Congruency: Quarterly revenue figures align across different reports.
• Cohesion: Customer IDs are consistent across sales and support databases.

As I progress through the book, I'm eager to explore how these Data Quality Specifications can refine my own data processing workflows.