The Observer: Data quality is best served with GenAI & why tech trends are like avocados

Hi folks,

We hope this newsletter finds you well. And by well, we mean full of turkey, mashed potatoes, and a whole lot of pie.

It’s Thanksgiving-time over here in the United States, which means taking some time to remember all the things we’re grateful for… like the time when our Airflow jobs didn’t fail… or hearing a stakeholder say ‘thank you’... oh, and the latest edition of the Observer, of course.?

Now that you’ve filled up on turkey and mashed potatoes, you can kick back and peruse the latest in data quality and beyond, like:

• ?? The latest data quality tips for busy data leaders?
• ?? This month’s hot, medium, and mild takes on GenAI
• ?? Info on our upcoming events, like AWS re:invent and IMPACT World Tour
• ??And our meme of the month

What’s new??

What we’re writing about:

• IMPACT 2024 Keynote Recap: Product Vision, Announcements, And More. In case you missed it, the IMPACT Data Observability Summit happened last week! We heard from exceptional speakers, including historian and creator of the 13 Keys to the White House presidential prediction model Dr. Allan Lichtman, U.S. Olympian Allyson Felix, Microsoft Fabric CTO Amir Netz, and more. Plus, we announced some pretty cool GenAI features – check it all out.
• A 5-Step Incident Management Framework for Enterprise Data Organizations. Incident management processes are like snowflakes: each one is a little different (depending on your organization’s structure, data needs, etc). We compiled a few best practices from leading enterprise organizations to build a framework that can serve as the foundation for your process.
• Scaling Data Quality Management for Business Rules & Baseline Rules. It’s important to consider the different types of data quality checks, including business rules and baseline rules, and how to use them effectively to scale your data quality management. We put together a quick primer to help you do just that.

What we’re talking about:

• Monday Morning Data Chat: Bridging the Gap Between Data Engineers and Analysts, and more. Matthew Mullins, CTO Coginiti, joined Joe Reis and Matthew Houseley’s podcast for an interesting episode about bridging the gap between data engineers and analysts, small data, the commoditization of data warehousing, and more.
• It’s Time to Merge Analytics and Data Engineering (Again) Chris Riccomini recently sat down with Josh Gray at the Artemis Blog to discuss data stack fragmentation, the future of dbt, cost savings, and more. Don’t miss their hot takes on the commoditization of data pipelines and the future of analytics engineering.
• This Week in Startups: Battling Data Downtime: Lior Gavish, CTO at Monte Carlo, joined Alex Wilhelm to talk about bringing data observability into the era of AI. From what he said, it sounds like a pretty good idea.

What’s hot? ???

We share one hot take, one medium take, and one mild take on what’s happening in the data space. Can you handle the heat?

• Hot: Tech trends are like avocados. At least that’s what Monte Carlo CEO Barr Moses told Jyoti Bansal in a recent CEO Rapidfire interview. What do you think?
• Medium: Data engineers and SWEs will merge. Zach Wilson of DataExpert.io shared an interesting take on how GenAI will make data engineers more efficient – and how data engineering and software engineering could merge in the AI future. Hm…
• Mild: Decentralized data is like sailing on choppy waters. J.P. Morgan shared an interesting take on the power of consistent data for streamlining investment decisions. Not too spicy, but it will add some nice flavor to your regular reading.

Let’s meet up!

Every season is data events season – and that’s just the way we like it. Catch us here next:

• IMPACT World Tour: While our virtual IMPACT event has passed, our IMPACT World Tour is in full swing! We’re headed to Los Angeles, Denver, and Minneapolis next. See you there?
• AWS re:Invent: What happens at AWS re:invent stays… in our booth scan list until we email you after. ?? Mark your calendars now for AWS re:Invent December 2-6. Be sure to stop by booth #667!
• The AI Summit NYC: We never miss an opportunity to talk about generative AI, so of course we’ll be at The AI Summit on December 11-12 in New York City.

What we’re reading:

What’s better than working with data? Reading about it! Here are a few articles from across the industry that piqued our interest.?

• AI’s Data Tsunami: Why Your Data Stewardship Needs an Overhaul. An interesting article in CIO about the tools and training data stewards need to get prepared for AI.
• Decoding the Hype: How to Effectively Integrate GenAI in Your Analytics Strategy. Ever wondered how higher education is leveraging GenAI? This article gives a look behind-the-scenes.?
• Monte Carlo adds GenAI to data observability. Check out our new GenAI-powered monitor recommendations, integrations with Microsoft Fabric, Databricks Workflows, and Informatica.? ?

Just for Fun

Stay tuned for more updates from The Observer in December!