LinkedIn Engineering的动态

LinkedIn has a massive plan to adopt the #Iceberg table format. Today we are revealing one piece of the puzzle: LakeChime, a data trigger service that is compatible with Hive and modern table formats such as Iceberg. LakeChime supports two main data trigger semantics: partition triggers and snapshot triggers, uniformly across Hive and Iceberg. We show how we integrate LakeChime within our data lake, and present a case study for integrating LakeChime with #Airflow to deliver an end-to-end user experience for incremental compute with #Spark. Enjoy reading the blog post! With Sandeep Dhillon, Nagarathnam muthusamy, Limian (Raymond) Zhang, Zhe Liu, Zhuo Wang, Trevor DeVore, Swathi Koundinya, Abhishek Tiwari, and Kamal D..

TechTarget is right on target, naming data lakehouses as a top trend in data management for 2024 - and Onehouse as a leader in data lakehouse interoperability through the OneTable (now Apache XTable (Incubating)) effort. TechTarget has been an early leader in spotting the importance of the lakehouse, linking to their recent article as to how the data lakehouse is the best fit for modern data needs. Check it out… https://lnkd.in/gyVAq3tq #onehouse #opensource #nolockin #datalakehouse #dataengineering #apachehudi #XTable

?????????????????? ????????!?? ?????????? ???????? ???? ?????? ???????????????????? ?????? ?????? #?????????????????????????????? ???????? ?????????????????? ???? ??????????????????? ?????? ?????? ???????? ???????????????????? ????????????????, ?? ???????? ????????-?????????????? ???? ?????? ?????????? ???? ?????? ???????? ????????????????????!??? ????, ???????? ???? ?????????? ???????? ??????????????????? It is an ???????? ???????????? ?????????????? ?????????? that provides reliability to the ???????? ??????????. And it's benifits, let's check- 1.>???????? ????????????????????????: Delta Lake is reshaping the landscape of data handling, bringing robust ????????(??????????????????, ??????????????????????, ?????????????????? ?????? ????????????????????)?transactions to Apache Spark. With features like schema enforcement and time travel capabilities, it ensures unparalleled data integrity and reliability. ???? 2>?????????????? ?????????? ?????? ??????????????????: Say goodbye to data silos! With Delta Lake, seamlessly manage both batch and streaming data, while optimizing performance through advanced compaction techniques. 3>???????????? ??????????????????????: Delta Lake enforces ???????????? ???? ??????????, ensuring that data adheres to a predefined schema ???????????? ???????? ???????? ?????????? ???? ???????????? ???? ????????.?This helps maintain data consistency and reliability. 4>???????? ???????????????????? ?????? ???????????????????????? : Delta Lake treats metadata just like data. Delta Lake can handle petabyte-scale tables with billions of partitions and files at ease, hence, it ??????????’?? ???????????? ???????? ?????????? ???????? ?????????????? ???????? ????????. 4>???????? ???????????????????? ?????? ??????????????????:?And the cherry on top? ???????????????????????? ???????? ???????? ???? ???????????????? ???????? ???????????????? ?????? ?????????? ???????????????????? ???? ???????????? ????????????????.???? Join me on this journey as we explore the latest developments in data management, AI, machine learning, and more.?? ????'???? ???? ?????????????? ?????????? ???????????????? ???? ?????? ????????, ???????????????????? ????????????, ?????? ????????????????????!??? Sumit Mittal Databricks Deepak Goyal Sunny Savita ??#MicrosoftFabric #DeltaLake #BigData #DataManagement #ApacheSpark

Resolution for #Serialization issue in #Spark : : : : #Bigdata Talks with Karthik K. & Seekho Bigdata Institute ???????? ?????? ?????????????? ???????????????????? ?????? ?????????????????? ?????????????????????????? ???????????? ???? ??????????: ??)?????? ???? ?????????????????? ?????????????????????????? ????????????: ? As mentioned before, using a more efficient serialization format, such as Kryo, can significantly improve performance. ??)?????????????????? ????????????????????????: ? Make sure that any custom classes that you are using in your?? Spark code implement the Serializable interface. ??)???????????? ?????? ???????????? ??????????: ? If your data includes large object graphs, try to flatten the object graph to reduce the amount of data that needs to be serialized and deserialized. ??)?????? ?????????????????? ?????????????????? ??????????????????????: ? Use broadcast variables judiciously to cache read-only data, but be mindful of their memory usage to avoid serialization issues. ??)?????? ???????????????????? ???????? ???????????? ???????? ??????????????????: ? Use operations such as reduceByKey and aggregateByKey when possible to reduce the amount of data that needs to be shuffled, improving performance. ??)???????????????? ?????????????????? ?????? ?????????????????????????? ??????????????: ? If you are using a serialization library such as Kryo, make sure to configure it properly to ensure that it is used efficiently. ??)?????? ??????????-???? ???????? ??????????: ? Use built-in data types such as Map, Array, and Struct, when possible, to avoid the complexity of custom complex data types. ??)?????? ??????????'?? ??????????-???? ????????????????????????: ? Spark has some built-in optimization such as column pruning, predicate pushdown, and auto-broadcast join to reduce the data transferred over network and thus improving the serialization performance. ??)?????? ??????????????: ? Caching the intermediate results of your computation can help in reducing the re-computation of data, thus reducing the serialization process. ???????? ???????? ???? ??????????????, ?????? ?????????????????? Tagged for Better Reach : Sivakumar Babuji Tushar Desai Ravi Kumar Soni Chidanand Tripathi Anand Vaishampayan Rajesh Kumar Maheshpal Singh Rathore Alvin Foo #DataEngineering #AlgorithmicTrading #BigData #FinancialData #DataAnalytics #MarketAnalysis #QuantitativeFinance #DataScience #Fintech #TradingTech #DataOps #MarketData #DataInfrastructure #TradingStrategies #DataPipeline

The month of June 2024 has been particularly confusing for data and AI professionals, especially those trying to understand the evolving landscape of data catalogs and metastores. With Apache Iceberg gaining momentum, Snowflake’s polaris catalog being open-sourced, unity catalog also being open-sourced, and Databricks acquiring Tabular, the situation has only grown more complex. In this article, I aim to demystify these developments. Take a read and get clarity. #datacatalog #mestatore #datamanagement

Another big win for data engineering and open source! GS #DataEng and #Legend continue to be at the forefront of #Data + #AI via integration with open formats and engines! #dataengineering #opensource #LegendForTheWin https://lnkd.in/e4_hNgvF

?? **Unlocking the Power of Time Travel with Delta Lake in Databricks** ?? In today’s data-driven world, the ability to track and manage changes in our datasets is crucial. Delta Lake, integrated beautifully with Databricks, offers an exciting feature:?Time Travel. This allows us to query previous versions of our data, enhancing our data governance and operational flexibility. ?? Time Travel: Querying Historical Data With Delta Lake, we can leverage two powerful methods to use time travel: 1.Using Version Numbers: Each transaction in Delta Lake is logged, and every modification generates a new version. This means we can easily query the state of our data at a specific version. For example, to access a dataset from version 1, we can use: SELECT * FROM my_table VERSION AS OF 1 2.Using Timestamps: Delta Lake also supports querying data as it existed at a specific point in time. This is particularly useful for auditing and compliance. Simply specify a timestamp: SELECT * FROM my_table TIMESTAMP AS OF '2024-08-13T12:00:00' ?? Rollback Operations Delta Lake makes rollbacks straightforward and efficient, allowing us to revert to a previous state if needed: 1.Using INSERT Statements: If we need to restore previous records, we can easily re-insert data from a previous version, allowing for selective restoration based on our needs: INSERT INTO my_table SELECT * FROM my_table VERSION AS OF 1 WHERE condition; 2.Using Restore Statements: For a complete restoration to a specific version, the RESTORE command simplifies the rollback process: RESTORE TABLE my_table TO VERSION AS OF 1; ?? Conclusion The ability to perform time travel and rollback with Delta Lake in Databricks empowers data engineers and analysts to be more agile and confident in their data operations. **Attaching scenario based examples** Are you leveraging Delta Lake's time travel features in your projects? Share your experiences! #DeltaLake #Databricks #DataEngineering #TimeTravel #DataGovernance

#databricks #sortaggregate #hashaggregate 1. Sort Aggregate: Operation Explanation: Sort Aggregate involves sorting the data based on the grouping column(s) before applying the aggregation function. This ensures that rows with the same grouping values are adjacent to each other in the sorted order. Pros: Well-suited when there is a need to retrieve data in a sorted order. Can benefit certain types of queries that require sorted output. Cons: Sorting can be an expensive operation, especially for large datasets. Performance may degrade as the size of the data increases. May require additional resources and time compared to hash-based approaches. 2. Hash Aggregate: Operation Explanation: Hash Aggregate involves using a hash function to partition the data into buckets based on the grouping column(s). Aggregation is then performed independently within each bucket, and the results are combined. Pros: Typically more efficient for large datasets as it avoids the need for a global sort. Can provide better performance in scenarios where sorting is not essential. Cons: May not be as effective when the query requires sorted output. The hash function and partitioning might introduce overhead, but this is usually outweighed by the benefits in terms of scalability. Performance Considerations: Dataset Size: For smaller datasets, the performance difference between Sort Aggregate and Hash Aggregate may not be significant. However, as the dataset size increases, the advantages of Hash Aggregate often become more pronounced. Query Complexity: The nature of the query can influence which aggregation method is more efficient. Queries that benefit from sorted output may perform better with Sort Aggregate, while others may benefit from the speed of Hash Aggregate. Resource Utilization: Sort Aggregate may require more memory and computational resources due to the sorting operation. Hash Aggregate, on the other hand, is often more scalable and can efficiently handle larger datasets. Query Optimization: The Spark SQL engine in Databricks performs query optimization and can choose the most efficient physical plan based on the query and underlying data distribution. In summary, the choice between Sort Aggregate and Hash Aggregate in Databricks depends on factors such as the nature of the query, dataset size, and the need for sorted output. It's recommended to analyze the query execution plans and leverage Databricks monitoring tools to make informed decisions regarding performance optimization.

?? Unlocking the Power of Adaptive Query Optimization (AQO) in Spark 3! In the fast-paced realm of data analytics, the ability to adapt and optimize queries on-the-fly is a game-changer. Let's delve into how Spark 3's Adaptive Query Optimization (AQO) is revolutionizing query performance, addressing dynamic data challenges, and propelling data processing efficiency to new heights. Spark 3 introduced Adaptive Query Optimization (AQO) to address several key issues in query optimization and execution. Here are three main issues that AQO helps solve: ??1. Dynamic Runtime Optimizations: Traditional query optimizers generate execution plans based on static statistics collected before execution. However, these plans may not be optimal for all runtime conditions. AQO adapts execution plans dynamically based on runtime statistics like data skew, join reordering, and predicate pushdown. This adaptiveness improves query performance in dynamic workloads where data distributions or query characteristics change over time. ??2. Skew Handling: Data skew refers to imbalanced data distributions, where some partitions or keys contain significantly more data than others. This can lead to performance bottlenecks in distributed processing frameworks like Spark. AQO addresses skew by dynamically adjusting join strategies, such as choosing broadcast joins for skewed partitions or using alternative join algorithms to handle skewed data more efficiently. ??3. Join Reordering: The order in which joins are executed can significantly impact query performance. Traditional optimizers use static join orderings based on statistics, which may not always be optimal. AQO leverages dynamic feedback during query execution to adapt join orderings based on actual data sizes and distribution, improving join performance and reducing overall query execution times. By addressing these issues through dynamic runtime optimizations, skew handling, and adaptive join reordering, Spark 3's Adaptive Query Optimization enhances query performance and scalability in diverse and evolving data processing scenarios. Thank you Sumit Mittal Sir for helping me understand this in such depth. #DataEngineering #BigData #DataAnalytics #Spark3 #AdaptiveQueryOptimization

Discover my latest post on Spark query optimization with Predicate Pushdown at Simplify Data Engineering ????