Prefect

We’re so grateful for Prefect—not just for simplifying our workflow orchestration but also for turning our fur babies into orchestration pros! ???? Thanks to their new Prefect goodies, our pups have mastered the art of life orchestration: automating snack delivery, optimizing squirrel-chasing routes, and monitoring belly rub SLAs. ??????. Oh, and they’re especially thrilled that everything is written in the "Paw-thonic" way! ?? Who knew workflow orchestration could be this pawsome? Thank you, Prefect, for keeping our work and play on point! ????

We’re so grateful for Prefect—not just for simplifying our workflow orchestration but also for turning our fur babies into orchestration pros! ???? Thanks to their new Prefect goodies, our pups have mastered the art of life orchestration: automating snack delivery, optimizing squirrel-chasing routes, and monitoring belly rub SLAs. ??????. Oh, and they’re especially thrilled that everything is written in the "Paw-thonic" way! ?? Who knew workflow orchestration could be this pawsome? Thank you, Prefect, for keeping our work and play on point! ????

?? Four game-changing trends are reshaping how we build modern systems, according to infrastructure veteran Chris Riccomini at Prefect Summit. After 20 years engineering at PayPal and LinkedIn, his insights are worth noting. The most surprising? The radical shift toward "Zero Disk Architecture" - completely abandoning local storage in favor of solutions like S3. This could fundamentally change how we think about data integration. Key transformations happening right now: - Traditional databases are being decomposed into reusable components, opening doors for specialized solutions - PostgreSQL is experiencing unprecedented growth, becoming the Swiss Army knife of databases - Durable execution frameworks are evolving to solve the exactly-once semantics challenge in distributed systems Want to understand how these trends will impact your system architecture decisions? Check out the blog post below that explores each trend and its implications for the future of infrastructure. What infrastructure trends are you seeing in your organization? Share below ??

The next stop on our IMPACT World Tour is less than 2 weeks away! Join us and our friends at Prefect, Fivetran, Sigma Computing, and Dagster Labs on December 3rd in Los Angeles. ?? Enjoy dinner and drinks, and hear an expert panel from Dana Neufeld, Data Product Manager at Fundbox, who will share her insights on driving trust in data and AI. Don't miss this! Register here: https://lnkd.in/dZsgeB9E #IMPACT #dataobservability #AI #datatrust #LAdataleaders #dataproducts

Struggling with orchestration costs and complexity? Endpoint was too—until they found Prefect. The results after switching from Airflow? - 73.78% reduction in orchestration costs - 78 new pipelines built in one quarter - Migration completed in under 2.5 months - Unified environment for ML and data teams Their ML and data engineering teams now operate seamlessly in a single environment—using the tools they rely on (like Docker, Kubernetes, Celery, and EKS) without the complexity that held them back before. Here's what Sunny Pachunuri, Data Engineering and Platform Manager at Endpoint, had to say: "Prefect's support feels like a true collaboration, significantly increasing our team's efficiency and satisfaction." Ready to transform your data orchestration? Read the full case study below. #DataEngineering #MLOps #DataOrchestration #Prefect

?? How does Rent The Runway manage thousands of designer items across our rental platform? At the recent Prefect and dbt Labs joint event, Vishal K., Staff Engineer at RTR, shared an inside look at their data infrastructure. The scale of RTR's operation: - 1,800+ data models - 30-40 dbt projects - All orchestrated through a single Prefect implementation A highlight of the stack: Business teams input data directly through Google Sheets, which automatically validates using dbt-expectations before flowing into production models. This democratizes data access while maintaining quality. The foundation? A powerful combination of Prefect + dbt + Snowflake, with parameterized flows orchestrating it all. Discover the detailed breakdown of RTR's data stack implementation, including migration strategies and key learnings that could benefit your team. Read the full technical deep-dive below. How does your organization handle data orchestration? Share your experience below ?? #DataEngineering #dbt #Prefect #ModernDataStack

?? Python developers, your input is needed! The official Python Developers Survey has launched its 8th annual edition. This crucial initiative helps track how the Python development landscape evolves year over year. ?? Why participate? Takes just 10-15 minutes Shapes understanding of the Python ecosystem Contributes to the broader Python community The survey results serve as a vital source of insights into the state of Python development worldwide. ?? Support this initiative here: https://lnkd.in/gtdQt9Cc

Dean’s List #21: Big Data London Spotlight?—?Prefect’s Workflow Orchestration Revolution: A Conversation with CTO Chris White ? At Big Data London, I had a fantastic conversation with @Chris White, CTO of Prefect, about how their Python-based orchestration framework is transforming how data teams handle complex workflows. Here’s why Prefect is becoming a favorite for data practitioners: What Makes Prefect Stand Out? Scalable Performance: Built to handle thousands of workflows and tasks with minimal overhead. Dynamic Workflows: Define workflows at runtime, enabling conditional branching and adaptability in native Python. Enterprise-Grade Security: A federated architecture supports secure, distributed operations without exposing infrastructure. Why Teams Choose Prefect? - Perfect for high-volume workflows that require scalability. - Ideal for teams building complex, dynamic workflow structures. - Trusted by distributed enterprises to balance collaboration and security. Prefect empowers teams to automate, recover gracefully, and optimize workflows—all while staying in the familiar Python environment. Big thanks to Chris for the insights! Jeremiah Lowin Sarah Moses Brad Evans Thomas E. Prefect #BigDataLondon #DataEngineering #WorkflowOrchestration #Python

?? What if your data pipeline fails halfway through processing critical financial reports? Without proper safeguards, a simple retry could create duplicate records and corrupt your data. Here's the reality: Data engineering teams often have limited control over input data but full responsibility for output accuracy. The solution? Idempotent data pipelines. Here's why they're game-changing: - Guaranteed consistency: Same input = same output, every time - Safe retries: No data duplication or corruption when failures happen - Built-in recovery: Resume from checkpoints instead of full reruns The most impressive part? Organizations using idempotent pipelines are seeing near-zero data inconsistencies in failure scenarios, compared to traditional approaches. Want to learn how to implement these resilient patterns in your data infrastructure? Sarah Krasnik Bedell breaks down the exact implementation strategies with real Python examples. Read the full blog post below. What challenges do you face with data pipeline failures? Share below ?? #DataEngineering #DataReliability #DataPipelines

Supply chain security is an ever present consideration for enterprises - it's great that PyPI now supports digital attestations! Prefect is one of many packages that have already been leveraging digital attestations beginning with our 3.1.0 release in October. For more details check out PyPI's blog below. #PyPi #Python