Datafutures

Day two at #dbt #coalesce2024 was great.. ?? Started with a fantastic talk by Allie K. Miller, and amongst her many great insights her vision of Autonomous Data and Analytics Development.. .. and then I got a chance to attend a few workshops covering dbt features I haven't played with yet.... 1?? dbt Mesh Explorer features new and amazing cross-project lineage that goes from project level all the way thru column level lineage ?? ... Engineer types tend to shy away from explorer, but I'm telling you this is a killer tool for all! 2?? Cloud CLI - maybe i'm late for to the game, but for all you dbt core fans who are missing the local IDE experience in cloud, should definitely check this out.. Super easy to setup and offers many controls over dbt development, and of course the full git command set ? ! The power user plugin for visual studio is pretty nice too... Oh!- and don't forget my session is tomorrow both IRL and online: https://lnkd.in/equVEezY #datamesh #analyticsengineering #dataengineering dbt Labs Datafutures

I'm excited to share that I'll be speaking at dbt Labs Coalesce this October. This conference is a great learning experience, especially in-person in Las Vegas. https://lnkd.in/eK6jFEZX I hope to see you there! #dataengineering #bigdata #analytics

Today we had a great lunch-and-learn with some of the greatest minds in data, and my favorite people, on Data Monoliths.? We literally had the collective knowledge of 100’s of projects from startup to large enterprise, and every industry on tap.? ?I wanted to share a few highlights. ?? Common Data Monolith disadvantages. ? 1) Large and complicated code bases hurting productivity (often only a single developer can change the code), 2) single point of failure, 3) dependency and merge conflicts ?? Data Monoliths are common and not always bad.? ? If you're a startup or a small organization building anything but a Data Monolith is probably premature optimization. ? We all agreed there is a difference between a well designed and poorly designed monolith. ?? Applying microservice patterns to anything can have its own pitfalls. ? It requires investment in tooling (more on that) and discipline, and just as importantly organizational maturity. ? Otherwise you might avoid a monolith, but you really have spaghetti! ?? Generative AI might make Data Monoliths more tolerable, or fix them!.? In the future we might develop data co-pilots to make everyone as efficient as that "single developer", and enable rapid refactoring of gangly code bases. ? However several were concerned that the general maturity of data engineering will hurt the ability to train such models. ?? Data Products are the new Microservices.? Independent, purposeful data products feel like the best way forward. ? We also couldn’t help but reminisce over their ancestral approach, the good ol’ Data Mart. ?? Tooling is Key. ?We need to take cues from the microservice world, invest in infrastructure tooling.? We need a /backstage.io for data! ?? Monolith or not, contracts matter.? We all agreed that no matter how a system gets architected, contracts are key.? ? Thanks to Marina Cruceru, Jeff Polack, John Hofmann, Bill Walrond, Sergey T., Eric Linden, Ramin Ostad, Inder Singh for spending their lunch chatting tech! Datafutures The Data Consortium #datamesh #datawarehouse #analytics #bigdata