Estuary

Interested in moving data into Microsoft #Fabric? Check out this video for a quick tutorial and get started in just a few minutes using our no-code pipelines: https://lnkd.in/dqzK6STd

We've just upgraded our #Iceberg connector! 1?? It now integrates with various new REST Catalogs, such as Apache Polaris and the Snowflake Open Catalog. 2?? Highly requested feature: The connector can now execute MERGE queries into your Iceberg tables, allowing you to have exact replicas of your source data that are updated in real-time. ?????? Read the full release announcement in the comments ??

When your data pipeline costs more than it should—and still doesn’t work right. Headset, a leading data and analytics company, needed a cost-effective, reliable way to move data from SQL databases into #Snowflake. ?? They tried one tool: costs kept climbing. ?? They switched to another: syncs lagged, records went missing, warehouse costs spiked. Then they tried out Estuary. ? 40% lower Snowflake compute costs ? 100% data integrity. No more missing records ? Near real-time ingestion instead of unpredictable job runtimes ? Reliable support & seamless experience The results speak for themselves: No more firefighting data pipeline issues. Just real-time, accurate analytics. Scott Vickers, CTO at Headset, put it best: "Estuary has been a game-changer for Headset’s data infrastructure. Compared to our previous solutions, it has dramatically improved reliability while reducing our overall costs significantly. The real-time ingestion capabilities ensure that our analytics are always powered by the freshest, most accurate data without the operational headaches we faced before." ?? Read the full story in the comments.

?? Real-time flight analytics at scale – without managing Kafka clusters Processing real-time air traffic data requires a scalable, efficient pipeline. In her latest technical tutorial,?Ruhee Shrestha?explains how to stream flight data from the?#OpenSky API into StarTree using #Estuary for sub-second analytics on live aircraft movement. The combination of Estuary and StarTree creates a powerful real-time data pipeline that simplifies streaming without managing Kafka and delivers sub-second analytics on high-velocity data. Here are some cool use cases you learn about: ? Tracking live aircraft positions and velocities ? Monitoring flight departures, delays, and anomalies ? Analyzing altitude changes and potential deviations ? Supporting real-time air traffic monitoring and decision-making Article link in the comments ??

?? Is Apache Iceberg the New Hadoop? Navigating Modern Data Lakehouses ?? In the ever-evolving landscape of data engineering, Apache Iceberg has emerged as a promising open table format, addressing challenges like data consistency and schema evolution. But is it the ultimate solution, or are we revisiting complexities reminiscent of Hadoop? ?? In our latest podcast episode, we sit down with Daniel Palma, Head of Marketing at Estuary and seasoned data engineer, to delve into this very question. We explore Iceberg’s potential, its operational challenges, and the intricate ecosystem it inhabits. From the “small file problem” to the chaos of catalogs, we uncover what organizations must consider before diving into Iceberg adoption. Key Takeaways: ? Operational Complexity: While Iceberg offers advancements, it introduces new challenges in metadata management and maintenance processes. ? Ecosystem Dependencies: Effective implementation requires a robust surrounding ecosystem, including catalogs and compatible compute engines. ? Strategic Adoption: Before embracing Iceberg, organizations must carefully evaluate their needs, resources, and operational capabilities. Join us for this insightful conversation to understand whether Apache Iceberg is the future of data lakehouses or if it’s a déjà vu of Hadoop’s complexities. ?? Listen now: Substack: https://lnkd.in/gaQgYFai Apple: https://lnkd.in/gMtzemFT Spotify: https://lnkd.in/gKmrcUjz Youtube: https://lnkd.in/gDCap38p #DataEngineering #ApacheIceberg #DataLakehouse #Hadoop #Podcast #DataManagement

#DuckDB is great for embedded analytics, but what happens when your data grows, or you need to integrate it with other systems? This article by Emily L. walks through a practical setup using: 1?? MotherDuck for a cloud-hosted, scalable DuckDB environment. 2?? Estuary to stream change data capture (CDC) events into DuckDB without manual ETL work. It covers setup, data loading, and query optimization, which is helpful if you're exploring DuckDB for analytics at scale. In just a few minutes, you can build your own real-time, no-code dataflow! Check out the full article in the comments ??

I’m working on a case study with a customer, and they shared some pretty crazy cost savings after switching to Estuary. Not only did they save a bunch on the data integration platform itself by moving away from Airbyte, but they also cut their Snowflake credit usage by 75% (!!!) thanks to our optimized ingestion process. Huge savings and one very happy customer! Full case study coming soon. Picture is from their Snowflake account after running the 2 pipelines in parallel for a while ??

It was awesome working with the team over at Estuary, we could not accomplished what we did without them. We are now in a great place long term with our data platform.

Join Daniel Palma and Benjamin Rogojan this afternoon for a discussion about #ApacheIceberg and its real-world challenges and best practices. event: https://lnkd.in/d4jRJUiq

Learn how Forward reduced their real-time analytics costs by 50% using Estuary. When Rockset deprecated its service, Forward, a fintech leader in embedded payments, needed a cost-effective real-time analytics replacement. Traditional ETL tools were either too expensive or lacked the required transformation capabilities. The Challenge: Managing real-time data ingestion from DynamoDB, MySQL & Snowflake Handling complex transformations for a single-table DynamoDB design Improving PostgreSQL (AWS Aurora) performance under heavy queries The Solution: ?? Estuary Flow provided real-time transformations, many-to-many data routing, and decoupled ingestion from the warehouse. Now, Forward seamlessly streams data to #PostgreSQL, MotherDuck, and #BigQuery while optimizing query performance. The Results: ? 50% cost reduction in real-time analytics ? Faster queries by shifting transformations upstream ? Simplified data integration with native connectors ? Future-ready stack, enabling AI use cases seamless migration to #ApacheIceberg ?? "Estuary powers real-time pipelines across our operational, analytics, and AI data stores. We unify CDC sources, extending our platform’s flexibility into the broader Kafka ecosystem." - Alexander Mays, Principal Engineer, Forward Forward now powers real-time dashboards, partner APIs, and AI analytics at half the cost. ?? ?? If you're looking to cut data costs while keeping real-time speed, let's talk. Full success story available in the comments!