Data Council

?? New Blog: Must-read data articles for November 2024's Top Ten This month, Lindsay MacDonald from Monte Carlo asks a critical question: Is data ready for GenAI? While AI seems ready to take off, are our data foundations prepared? Let’s find out: https://lnkd.in/gNNkTk59 Mission Lane: Built an always-on AI compliance testing system for fintech Confluent | Adam Bellemare: How to handle bad data in event streams Yelp: Optimizing Redshift data loading with DBT & Spectrum Fowler Martin | Kiran Prakash: Using fitness functions to scale data product governance Mikkel Dengs?e: How 40 top data teams structure their orgs Felicis | Astasia Myers Eric Flaningam: AI data infrastructure landscape and future bets Jack Vanlightly: Balancing incremental processing vs data quality Valliappa Lakshmanan: Why your architecture needs a "platinum" layer Barr Moses: Survey: 91% building AI but 2/3 don't trust their data ?? Ben Lorica 罗瑞卡: How leading AI labs prioritize data quality first Want to curate a monthly top ten? Contact us at: [email protected]

Last Call! Our Call for Proposals for Data Council 2025 ends tomorrow. We’re searching for the sharpest technical minds in data engineering, MLOps, AI and infrastructure to share their insights. Why submit? → Showcase your work to a global audience of top-tier engineers and founders → Network with talent from companies like Meta, Vectara and Anthropic → Get feedback on your ideas from industry veterans → Potentially catch the eye of investors (like me ??) If you’re building something groundbreaking, we want to hear about it! Apply here:?https://lnkd.in/gvEnaXPt Or if you know someone else who is, tag them in a comment so they see this post. ?? See you April 22-24 in the SF Bay Area.

The current cycle of AI tooling focuses too heavily on primitive chat use cases.?The next generation will be fundamentally different—requiring capabilities that mirror how people actually work: → Long-term planning abilities and multi-step reasoning to pursue complex goals → Persistent memory and knowledge management to learn from mistakes and develop skills → Tool-building capabilities to create and optimize solutions This shift from chat to semi-autonomous agent assistants represents a massive opportunity in AI infra. LLM apps are everywhere. The hard technical problems—and bigger opportunities—are in designing systems, knowledge bases, backend data workflows and tool automation. This is what powers serious enterprise AI. Ashwin Ramesh's talk at Data Council 2024 validated this thesis: AI's role is transforming from conversation partner to technical collaborator and autonomous agent. The infrastructure to enable this is being built now. If you're building AI infra or devtools, this talk is worth 15 minutes of your time. Ashwin digs into specific research projects tackling planning systems, memory retention and tool creation. Link to talk in comments.

Thank you to all our fantastic track hosts for thoughtfully guiding the programming as we prepare for Data Council 2025! ?? Your expertise in curating technical insights, real-world use cases, ML applications, and cutting edge AI applications into talk form are sure to make this upcoming Data Council the best yet as we return to the SF Bay Area! Special thanks to: Scott Breitenother Shawn swyx W Sean Taylor Carlos Aguilar Tristan Zajonc Daniel Francisco Sai Krishna Srirampur Maggie Hays Sean Anderson Karin Berkowitz (Wolok) ? Roger Magoulas By the way, we're extending the Call for Proposals deadline to November 22nd—submit your talk idea today to take the stage! https://lnkd.in/gBCXvaAz

Your boss calls it risky; we call it innovation. Share your work with our Data Council community. Data Council 2025 is now accepting talk proposals. We're searching for technical deep-dives that showcase the future of data engineering, AI systems and MLOps. If you’ve got insights on: → applied AI → OSS data tools → AI engineering → storage systems → computer vision → AI and ML tooling → data infrastructure → data science models → analytical processing → building strong data cultures → or any cutting-edge technical data topic …we want to hear it. Don't miss the chance to add your voice to the mix. Submit your talk ideas by November 15th here: https://lnkd.in/gvEnaXPt

Data Council 2025 Call for Proposals is open. We're looking for talks that showcase the next generation of data and AI infrastructure. Topics we're excited about: → Scaling challenges in AI pipelines → Novel approaches to data storage, quality and observability → Advances in AI deployment and monitoring If you're building tools that make data scientists and engineers more productive, we want to hear from you. This is your opportunity to share your work with a global audience of technical decision-makers. ?? Apply here: https://lnkd.in/gXATfv6W ??? April 22-24 ?? SF Bay Area Data Council

Cybercrime is expensive, with global annual costs estimated in the trillions. As one of the world's leading identity and access management platforms, Okta secures access to an enormous array of software tools and data. This makes it a juicy target. Okta's Defensive Cyber Operations team has to keep it all safe from attack: setting tripwires, embedding canaries, and processing trillions of rows of data in the hunt for cyber threats. At Data Council '24, Jake Thomas of the Data Foundations team at Okta walked us through the challenges of building their embedded OLAP (Online Analytical Processing) tooling to enable near-real-time access to data—a critical step in sniffing out malicious activity. Jake walked us through the evolution of their system: from the slow, expensive V1 architecture to a deep technical dive on their vastly improved V2. It's one of the most-viewed videos from this year's Data Council. Give it a watch: https://lnkd.in/eyPi7-8B Fantastic talk, Jake!

This is your chance to share work that's too cutting-edge for your day job ?? Data Council 2025 is now accepting talk proposals. We're searching for technical deep-dives that showcase the future of data engineering, MLOps and AI infra & apps. What we’re looking for: → Deep Technical Depth: We're looking for talks that dive into the latest advancements from a technical perspective. Bring your expertise from the cutting edge. → Real-World Challenges and Solutions: Share stories of overcoming complex challenges in building data platforms and AI products at your company. We’re interested in real-world applications, case studies and real examples of systems you've built.? → Innovative System Design: We value fresh, hands-on approaches to system design, especially when paired with thoughtful discussions on architectural tradeoffs. Share your unique methods and insights for building robust systems. → Evaluation Criteria: Talks are judged on technical depth, the speaker’s experience, and their potential appeal to our audience. ?? Submit your proposal: https://lnkd.in/gvEnaXPt See you April 22-24 in the SF Bay Area. Data Council

Don’t let your brilliant ideas stay locked in your codebase. The Data Council 2025 Call for Proposals is open, and we’re looking for the best and brightest to join us on stage. Whether you’re a data engineer with a clever solution or an AI researcher with fresh insights, now’s the time to share it with the world. Submissions close soon—get yours in before it’s too late! https://lnkd.in/dyA8qCks

Unveiling our Data Council 2025 tracks: → Data Engineering & Infra → AI Engineering → Data Science & Algos → Applied GenAI → Analytics & BI → Foundation Models → MLOps & Platforms → Databases → Data Culture → Lightning Talks We're covering the entire data ecosystem over three days of no-nonsense, data-driven discussions. ??? April 22-24 ?? SF Bay Area Now, here's where you come in. Calling all data wizards: We're hunting for cutting-edge technical talks that'll shape the future of data engineering, MLOps, and AI infrastructure. No fluff—just pure, unadulterated tech insights. Submit your talk ideas for Data Council 2025 by November 15th here: https://lnkd.in/gvEnaXPt