Monte Carlo

For data teams, nothing causes a bigger headache than when your pipeline fails and you don't know why - or how. All too often, we over-index on the symptom instead of focusing on curing the disease. This is a problem because: - It makes stakeholders happy in the moment (my dashboard is updated) but doesn't get to the root cause. - It incentivizes tech debt instead of building robust and scalable systems. - It turns firefighting into the de-facto method of incident management instead of defining clear owners and processes. - In the long term, it positions data as a cost center instead of a well-managed, reliable revenue driver. And the list goes on. But there's a better way, and my colleague Neil Gleeson, one of Monte Carlo's most tenured customer success managers, shares how. Check out his post on the core steps for rolling out an effective and and headache-proof data incident management program.

Don’t miss Bronte Baer, Manager of Data Platforms & Analytics Engineering at Earnest, at IMPACT 2024 on November 14th. ?? In her session at 11am, Bronte will share lessons learned during Earnest’s transition from a monolithic data system to a microservices-based architecture, including the reliability challenges that come with data migration, monitoring, and validation in a new decentralized environment. Mark your calendars! Register here: https://lnkd.in/eB8Gaeae #IMPACT2024 #dataobservability #datareliability #dataquality

Earlier this year, I was stressed out by a debt collection call. A small mistake was made by PG&E, where they misread my $184 check as $18.40. Just one decimal in the wrong place, and suddenly, I’m on a debt collector’s list, with my credit score at risk. This isn't just about my experience. Let's do some quick math: PG&E has ~16 million customers. So, even with a 0.1% error rate in their billing, that's potentially 16,000 affected people! Now, in a world where a misplaced decimal can spiral into a collection call, imagine the impact of data quality issues on AI products. Because when it comes to AI, the stakes are higher. A small crack could get to the algorithms and magnify when the model is in production. “One small piece of garbage in, 100X garbage out.” Don't let the data product you build make?the same mistake that sent me into a panic. If you want to learn how to safeguard your data and AI systems, the upcoming Data Reliability Summit by Monte Carlo is your go-to event. You'll learn from top data leaders about: · Driving trust in data and AI at scale · Building robust data systems for increasing AI adoption · Avoiding million-dollar mistakes with data quality issues Register here: https://lnkd.in/eBi4YEgc In the end, we all want our bills paid, our customers happy, and our AI trustworthy. Let's make it happen together by taking care of our data quality first.

The best piece of advice I ever received? Don’t listen to advice. Listen to your customers. That ethos has helped me both build a company - Monte Carlo - and a category - data observability as we know it today. Of course, like all "advice," it's easier said than done, particularly when the FOMO hits. But FOMO doesn't provide constructive product feedback or speak on behalf of your vision to future customers. FOMO doesn't build a category with you or help you solve real world problems. In my recent interview with Jyoti Bansal for the CEO: Rapidfire newsletter, I share some of my perspectives on FOMO, leadership, company building, and category creation. Check out the newsletter to learn more. Thanks, Jyoti Bansal, for the opportunity - had a blast! https://lnkd.in/g9MmaeMH

As a data analyst, delivering trusted insights to your stakeholders can cost a lot of time – and maybe even some sleep. ?? Many analysts lose hours profiling their data, identifying thresholds,?creating manual rules, and following up on data quality issues — all to make sure the data products they deliver meet?their data quality standards. But, every hour that’s spent on data quality is another hour that can’t be spent delivering value. So, what’s a data analyst to do? We rounded up five ways you can take data quality off your endless to do list and make it a little less painful, including: ? Automating test creation for baseline data quality rules ? Leveraging AI-powered monitor suggestions for business rules ? Deploying no-code validation tools ??Routing data quality alerts to your domain Slack channels ? Conducting regular data audits to prioritize coverage and deprecate unused tables Learn more here: https://lnkd.in/g7-8e_vx #dataanalytics #analyticsengineering #dataquality #dataobservability

Mark your calendars... IMPACT World Tour: Atlanta is coming up soon! ?? Join us and our friends at Dagster Labs on October 24th for an evening of drinks, dinner, and discussion around the critical theme of driving trust in data + AI. Plus, you'll hear from Zachary Lancaster, Data Engineering Manager at Warner Bros. Discovery, who will share his insights on implementing processes, technologies, and organizational changes to ensure his team is running on reliable data and AI. ?? Don't miss this! Register here: https://lnkd.in/e6iRcBVZ #IMPACT2024 #dataobservability #dataquality #AI #datatrust

Virgin Media O2's data democratization journey is truly inspiring. Leading the charge? Abigail Beaumont (Brockbank) and Bhagya Reddy. Check out this great interview about their journey to data, the criticality of data democratization and access for enterprise growth, and what's next for the Virgin Media 02 data team: https://lnkd.in/gH7feN8W

Join us for our IMPACT World Tour in London on November 5th! ?? Register here: https://lnkd.in/eTVcNUBc

If you haven't registered for IMPACT Summit yet, now's the perfect time. ?? We're giving away Zach Wilson's All-Access DataExpert.io subscriptions to four lucky IMPACT attendees! The subscription includes over 100 hours of self-paced educational content focused on data engineering fundamentals and best practices. Subscribers will learn how to use Astronomer, Apache Spark, Trino and more! Plus, they'll get a discount for his live bootcamp in January. What are you waiting for? Register for IMPACT here: https://lnkd.in/evJYkmec #IMPACT2024 #dataengineering #bootcamp #dataobservability #dataquality

Going Beyond Data Quality Rules for Better Business Insights For most analysts, delivering reliable insights means writing data quality rules –and lots of them – to monitor for issues and data quality dimensions. But, every second you spend writing and managing a data quality rule is another second you can’t spend delivering new value for stakeholders. But, what if—instead of writing all those rules—you could automate them instead? How might that impact your ability to deliver trusted business insights? Join us as we explore the challenges of data quality management, and how AI-powered data quality features can accelerate incident detection workflows to provide more visibility into the health of your data products.