Delta Lake

Join us for the first session of our new series, Ask the Authors! ?? This interactive series offers a unique opportunity to meet the minds behind Delta Lake: The Definitive Guide and get your questions answered live. ?? Tuesday, November 26 at 9AM PT ?? Denny Lee, Prashanth Babu, R. Tyler Croy, Tristen Wentling, & Scott Haines ?? LIVE on LinkedIn & YouTube If you're looking to enhance the performance of your data lake, tackle complex data challenges, or simply learn more about #DeltaLake's capabilities, this event is for you. Learn how Delta Lake provides an optimized storage layer that enhances streaming and batch data operations' reliability, security, and performance. PLUS, we're giving away 5?? copies of Delta Lake: The Definitive Guide!??? Don't miss this chance to interact directly with the experts, gain valuable insights, and connect with fellow Delta Lake users and contributors. RSVP today! ?? #opensource #oss #linuxfoundation #lfaidata

?? Happening tomorrow, November 26 at 9AM PST! Don't miss the first session of our brand-new series, Ask the Authors! ?? Meet the minds behind Delta Lake: The Definitive Guide and get your questions answered live! If you're looking to enhance the performance of your data lake, tackle complex data challenges, or simply learn more about Delta Lake's capabilities, this event is for you! ?? ?? Submit questions early: https://lnkd.in/eMqk8YzJ ?? RSVP TODAY: https://lnkd.in/eKZZwj2M PLUS: We're giving away 5?? copies of Delta Lake: The Definitive Guide! ?? cc Denny Lee, Prashanth Babu, R. Tyler Croy, Scott Haines, Tristen Wentling #opensource #oss #linuxfoundation #deltalake

?? Flushing 27+ GB/min from Event Hub to Delta Lake with delta-dotnet into OneLake! ?? Blog + Demo: https://lnkd.in/em969WhU ?? Learn how to deserialize 1.7 TB/hour into Delta Lake using C#/Dotnet, powered by highly concurrent APPEND-only flushes. Dive into the capabilities of the delta-dotnet Nuget package and native parquet writers to enable blazing-fast "Delta Bulk Loads." cc Raki Rahman #opensource #oss #linuxfoundation

?? When we talk about streaming, does it always mean real-time, or can we think of it in a slightly different context? Sometimes, real-time streaming isn’t necessary, but efficiency is still important. By processing data incrementally with streaming triggers, you can maintain resource efficiency while handling large volumes of data without compromising state management or analytical readiness. ?? Scott Haines & Denny Lee dive deeper into this topic: https://lnkd.in/edk_qswX #opensource #deltalake #oss #linuxfoundation #lfaidata #streaming

?? Ready to level up your Delta Lake knowledge? Join us for the first session of our new series Ask the Authors—a LIVE event where you can ask your #DeltaLake questions and get real-time answers from authors of Delta Lake: The Definitive Guide! Learn how Delta Lake boosts streaming and batch data operations with an optimized storage layer for better reliability, security, and performance. ?? ?? Tuesday, November 26 at 9AM PT ?? Denny Lee, Prashanth Babu, R. Tyler Croy, Tristen Wentling, & Scott Haines ?Submit your questions in advance: https://lnkd.in/ecYb4ryi ?? PLUS, 5 lucky attendees will win a free copy of the book! RSVP today. ? #opensource #oss #linuxfoundation #lfaidata #deltalake

With 10K+ Slack members, 190+ developers from 70 organizations, and 20M+ monthly downloads, Delta Lake is transforming how organizations manage data lakes. ?? As an LF AI & Data incubation project, Delta Lake delivers reliability, performance, and scalability for data-driven teams worldwide. ?? Join us TOMORROW, November 19 at 9 AM PT to learn why Delta Lake is a cornerstone for modern data engineering. Hear from Denny Lee, Principal Developer Advocate at Databricks, as he highlights the key features and innovations driving Delta Lake’s success. ?? Live on LinkedIn & YouTube ?? Hosted by Vini Jaiswal, LF AI & Data TAC Chair Don’t miss out—RSVP today! ? #opensource #oss #linuxfoundation #lfaidata

You don't have to optimize your Delta Lake table manually. Use Optimized Write or Auto Compaction to automatically merge small files. ? Optimized Write combines many small writes to the same partition into a single big write command. This is great when multiple processes are writing to the same partitioned Delta table. Auto Compaction runs a small `optimize` command after every write operation. Data from files under a certain threshold size is automatically combined into a larger file. You can use Optimized Write and Auto Compaction separately or together. It's up to you and your use case. You can run a regular VACUUM operation with Auto Compaction jobs to clean up the remaining small files and save storage space. cc Avril Aysha #opensource #deltalake #oss #linuxfoundation #lfaidata

`MERGE` is one of the most powerful Delta Lake operations? With Delta Lake merge you can apply multiple INSERT, UPDATE and/or DELETE operations in a single transaction. You can also specify conditional statements for your upserts. Merge operations allows you to make very precise changes to your data **without having to rewrite the entire table**. Updating or deleting a row is just a simple change to the transaction log. This is much more efficient than immutable file formats like Parquet. delta-rs has a great API that makes combining different conditional merge operations super easy. Here's how to do an UPSERT with delta-rs: Credit: Avril Aysha #deltalake #rust #deltars #linuxfoundation #opensource

Join us for the first session of our new interactive webinar series, Ask the Authors & connect with the minds behind Delta Lake: The Definitive Guide! ?? Learn about the optimized storage layer that powers your batch and streaming data operations, get expert tips on managing complex data challenges, and get answers to your questions—live! ?? Plus, we’re giving away 5 copies of Delta Lake: The Definitive Guide to attendees! Save your spot now! ?? https://lnkd.in/eKZZwj2M Submit your questions in advance ? https://lnkd.in/ehegFEaE cc Denny Lee, Prashanth Babu, R. Tyler Croy, Scott Haines, Tristen Wentling #opensource #oss #linuxfoundation #lfaidata

Trivia Time! ?? Curious about how Delta Lake got its name? ?? Just as a river delta gathers and enriches soil, Delta Lake gathers and enriches diverse data streams, centralizing them into a unified data lake. The term “delta” also reflects the incremental, streaming updates central to Delta Lake’s design, enabling efficient, real-time data processing. Watch the full Delta AMA session ?? https://lnkd.in/eA6iZCz4 cc Scott Haines, Dominique Brezinski, Denny Lee #opensource #linuxfoundation #lfaidata #oss