Data Reading Club #4

Testing Data Pipelines: The Modern Data Stack Challenge by Ari Bajo Rouvinen

How do you know that what you deploy to production will behave as expected and not break anything else? ??

This is something I have been struggling with these days and so naturally, articles covering testing ?? catch my attention.

In this article, Ari Bajo Rouvinen explores alternative approaches and technologies when testing data pipelines that spread across multiple layers.

Collaboration Just Got A Lot Easier by Jacob R?nnow Jensen

This article prompted me to try ChatGPT at work. Until now, I really tried to not fall for the hype and was at times overwhelmed by the amount of ChatGPT mentions on my LinkedIn feed.

Jacob R?nnow Jensen has shown me a potential of ChatGPT in my line of work, and, specifically when using Azure Data Factory to ingest data from API.

?? Takeaways from this article and after trying ChatGPT myself:

• ChatGPT can easily and quite precisely answer questions that you need quick answers to. The other day I used ChatGPT to learn about incremental loading and how to do it in Azure Synapse pipelines.
• It can save a lot of time on writing code and documentation
• It can serve as a translator between different roles with area of data

??

?#datareadingclub?series is a weekly LinkedIn newsletter aimed at sharing a short list of thought-provoking material that I came across that week. The list is not limited to reading material, but also includes podcasts, video clips etc.