Machine Learning

The Apache Hudi 1.0 release is now out! ?? It is a hallmark milestone for the community and a testament to the power of open-source collaboration. More than 60 engineers contributed code, and others helped in various ways, including documentation, testing, or feedback. Congratulations to everyone involved. Hudi 1.0 is the most powerful advancement for the data lakehouse community to date. It is rich with novel features and advanced new concepts that have never been possible before for data lakes. Read the launch blog to dive into the nerdy details to learn more about: ?? Secondary Indexing - 95% decreased query latency with additional indexes that can now accelerate queries beyond just primary keys. For example you can now build a Bloom Index on any column using new SQL syntax to create/drop indexes asynchronously ? Expression Indexes - Borrowing a page from Postgres, you can now even define indexes as expressions of columns. This allows you to build crucial metadata for data skipping without relying on table schema or folder directory structures usually needed when partitioning. ?? Partial Updates - 2.6x performance and 85% reduction in write amplification with MERGE INTO SQL statements that can now modify only the changed fields of a record instead of rewriting/reprocessing the entire row for massive performance improvements on update heavy workloads. ? Non-blocking Concurrency Control - NBCC enables simultaneous writing from multiple writers and compaction of the same record without blocking any involved processes. This is achieved with lightweight distributed locks and new TrueTime semantics ?? Merge Modes - First-class support for both styles of stream data processing: commit_time_ordering, event_time_ordering, and custom record merger APIs. ?? LSM timeline - Hudi has a new revamped timeline, which now stores all action history on a table as a scalable LSM tree so that users can retain a ton of table history. ? TrueTime - Hudi strengthens TrueTime semantics. The default implementation assures forward-moving clocks even with distributed processes, assuming a maximum tolerable clock skew similar to OLTP/NoSQL stores Read more and try out the new features for yourself here: https://lnkd.in/g-m9VBt7 #apachehudi #datalakehouse #opentableformat #dataengineering #apachespark #apacheflink #trinodb #awss3 #distributedsystems

If one thing has been made abundantly clear in 2024, the next hotspot in data to keep your eyes on is the evolution of the DATA CATALOG market. Now that open table formats have unlocked storage as interoperable, the next vendor lock-in battle will be fought with catalogs ?? After I held a panel discussion with the industry leaders building the next generation of catalogs, I started to do my own research. I found many great blogs and resources, but I quickly realized that there was no neutral or comprehensive source of information which compared all the features I cared about across offerings. After deep research, and spending time building with each catalog, I created a teardown with feature comparison matrices and I created a metastore to rank each data catalog across features ranging from access controls, data quality, data discovery, and much more. Ranking results from my research: ?? - Datahub ?? - Databricks Unity Catalog ?? - Atlan 4?? - AWS Glue 5?? - Unity Catalog OSS 6?? - Apache Gravitino 7?? - Apache Polaris As is usually the case, there is not a single one catalog is the best for every use case, so please do your own research. I guarantee I have some mistakes in this article, please help me find and fix them ?? READ THE RESEARCH ?? : https://lnkd.in/gkzEMw33 #datacatalog #unitycatalog #datahub #apachepolaris #atlan #awsglue #apachegravitino #dataengineering #apachehudi #apacheiceberg #deltalake

Announcing my new book - "Engineering Lakehouses with Open Table Formats" ?? TBH, I have been thinking about this for quite some time. A lot of times, in conversations with folks exploring table formats, questions have come up around choosing the right table formats, understanding use cases, and designing the overall lakehouse architecture. So, the goal with this book is to provide a comprehensive resource for data/software engineers, architects, and decision-makers to understand the essentials of these formats. But, also to elaborate on some of the less talked about 'core' stuff (beyond marketing jargons). Specifically, the book will target 4 angles: ?? Table format Internals - e.g. How ACID transactions works, What is a Storage Engine, Performance optimization methods, etc. ?? Decisions on selecting a table format - factors to consider from a technical standpoint, ecosystem, features. ?? Use-cases and how to implement - streaming/batch, single-node workloads, CDC, integration with MLFlow, etc. ?? What's happening next - Interoperability (Apache XTable (Incubating), UniForm), Catalogs (Hive to newer ones such as Unity Catalog, Apache Polaris (Incubating)) I’ve been fortunate to have first-hand experience working with open table formats like Apache Iceberg and Apache Hudi primarily, and in some capacity with Delta Lake (circa 2019). And, I intent to bring those experiences and touch upon the intricacies along with some of the pain points of getting started. I am also thrilled to have Vinoth Govindarajan as a co-author, who brings a wealth of experience building lakehouses at scale with these formats at organizations like Uber and Apple. We have drafted the first few chapters, but there's still work to do. We’d love to take this opportunity to learn more from the community about any additional topics of interest for the book. I'll be opening a formal feedback channel in a few days. Oh, and the book is already available for pre-order on Amazon (link in comments). Thanks to Packt for their continuous support in making this a solid effort! #dataengineering #softwareengineering

Unity Catalog vs Apache Polaris (Incubating)… It’s time for some Friday armchair analyses. Unity Catalog was open sourced from Databricks on 6/13 and donated to the LF D&AI Foundation on 6/20. Polaris Catalog was open sourced from Snowflake 7/30 and donated to the ASF Incubator on 8/21 Both of these projects are very exciting advances in the open source community and I look forward to both creating stronger solutions for data governance and ability to interoperate data across diverse tools. It is still very early days for both of these projects, but here are some interesting nuggets so far: ?? Table Metadata Format support: Unity Catalog (UC) = Apache Iceberg, Delta Lake, and Apache Hudi Polaris Catalog (PC) = Iceberg only ?? ? GitHub stars (go smash a star on both or your fav): UC = 2,156 PC = 960 ?? Github contributions (last 30days): UC = 31 authors | 49 Merged PRs?| 19 closed issues PC = 43 authors | 129 Merged PRs | 26 closed issues In 2024 we finally resolved table format wars just in time for the new battle of the catalogs? Which gameboy cartridge are you planning to plug into your data stack? I’ve personally tried one and planning to make a demo on the other soon. Has anyone else tried both and have thoughts to share? At Onehouse we will be supporting both. Good luck to cloud vendors though making sure their entire complex portfolio of products works with both plus their own catalogs they are building? #Snowflake #Databricks #UnityCatalog #PolarisCatalog #DataLakehouse

Bloom Filter in Parquet & Lakehouse Table Formats. Parquet filter pushdown is a performance optimization method that prunes irrelevant data from a #parquet file to reduce the amount of data scanned by a query engine. You can skip row groups typically using 2 ways: - column min/max statistics - dictionary filter ? Column statistics include minimum and maximum values that allow for range-based filtering. ? Dictionaries provide more specific filtering, enabling readers to exclude values that fall within the min and max range but are not listed in the dictionary. Problem with dictionary is that it can consume more space with higher cardinality columns. This results in columns with large cardinalities and widely separated min and max values lacking effective support for predicate pushdown. This is where the 3rd approach comes. ? A bloom filter is a probabilistic data structure that allows you to identify whether an item belong to a data set or not. ? It either outputs: "definitely not present" or "maybe present" for every data search. By using Bloom filters, you can efficiently skip over large portions of the Parquet file that are irrelevant to your query, reducing the amount of data that needs to be read and processed. In lakehouse platforms like Apache Hudi, users can utilize the native?Parquet bloom filters, provided their compute engine supports Apache Parquet 1.12.0 or higher. I also read an amazing blog by the Influx Data Team on Parquet's bloom filter implementation. Link in comments. #dataengineering #softwareengineering

The Data Guy, George Yates talks about where to use #ApacheHudi ??and how to get up and running fast! ? Follow along the Spark quick start ?

Interested in leveraging an open table format? Follow this short tutorial Thomas Hass and easily leverage your lakehouse across the existing vendor ecosystem ????

?? Discover the power of storing AI vector embeddings in a data lake for cost-effective storage and seamless integration! Learn how Apache Hudi enables you to manage large datasets efficiently and incrementally bring vectors to your target database. Read more: #AI #DataLakes #ApacheHudi #DataStorage

The dirty little tech secret in 2024. Most AI companies are service companies getting product valuations. 100x valuations on a Saas product. Cool. But this isn’t a Saas product. These companies are white-gloved consulting companies. Call a spade a spade. They masquerade as a product company and trojan horse a platform into the client's infra. I can’t be the only one who sees this, can I? The value doesn’t come from the AI companies ‘product’. The product is a commodity. The consulting is the value add. But hey, who am I to rock the boat? Let the good times roll.

How PENN Interactive uses Apache Hudi ????