How do you incorporate data quality feedback loops and continuous improvement in your data lake?

I have used S3/Spectrum for storage and AWS Redshift for data processing in my role. We saw descrepencies between Redshift and Spectrum. We added additional data integrity check to ensure data kepr its count ans structure comparable with the source.

Very good advice. DQ is subjective. If you're just getting started, you might want to focus on a set of high-value validity and accuracy rules to ensure the data for the project/use case does not have glaring deficiencies. Standard metrics like duplicates, invalid formats, and nulls can be caught with AI-based anomaly detection—when they deviate from the norm.

Establish the data quality standards that processes need to follow. Identify metrics to be used to measure data quality. Profile data using tools and analyze any issues. Cleanse data and ensure it meets data quality standards. Validate data against defined quality metrics. Implementation of feedback loop between those who work with data and who enter the data, this will enable to address reasons for poor data quality in long term.

Don't forget about data observability tools that can automate the monitoring of your critical systems. They are usually easy to configure—in some cases just a couple of checkboxes.

Usually all the blame for data quality goes to the data processors(ETL teams) instead of data producers. There should be a proper process in place for gathering feedback on data quality from both data producers and data consumers. Leadership should encourage collaboration between data producers and data consumers to collectively address data quality issues.

Establish the data quality standards that processes need to follow. Identify metrics to be used to measure data quality. Profile data using tools and analyze any issues. Cleanse data and ensure it meets data quality standards. Validate data against defined quality metrics. Implementation of feedback loop between those who work with data and who enter the data, this will enable to address reasons for poor data quality in the long term.

One thing I’ve noticed is data engineers forget about the metadata access we can use to understand our implementation decisions. An example of an innovative quality assurance feedback loop is to create a metadata-model containing metrics that then evaluate quality. This includes things like unique comparisons, null assessments and so on. Add that to an automated ETL and you’re able to get daily updates.

In my experience it was important to ensure that the data quality activities are prioritized at the beginning of the data asset on-boarding. It’s much more costly to perform these activities once the data has been productionalized. I found that having a solid data catalogue is of upmost importance to any data repository, and it solves more than one challenge - for example it creates an engagement opportunity for business to contribute to the data quality checks and their eventual automation. So delegate some tasks to business to contribute to data catalog and as a bonus you will get their DQ input.

A raw truth: The constraint capacity that storage give to developer depend the what storage you choose; The main benefit is that storage can act like defense layer in human error cases with this capacity. Depende the capacity, you can implement some data quality rules about it to previous to store the data. In general, data lake have not it; so you recreate it programmatically.

Another approach to DQ automation is to flow some of your DL data into a DW, where you design a good schema that prevents poor data from entering the warehouse. This works as a filter that traps DQ errors. Again, tight business engagement in this is very important: often people who run a data warehouse distance themselves from business so much that they cannot answer real-life business questions, so watch out for this.