Last Week in a Byte on Delta Lake | 2022-12-20

Now you can watch or read the latest #DeltaLake news a week late (2022-12-20 edition)!

Integrations

• aws-pandas-sdk (aka aws-wrangler) which includes an optional dependency to Delta Lake to simplify integration with #AWS services such as #Athena, #Glue, Neptune, #DynamoDB, #EMR, My SQL, #S3, and more: https://github.com/aws/aws-sdk-pandas/pull/1834?
• polars, blazingly fast DataFrames in Rust, Python, and node.js has support for reading Delta Lake tables: https://github.com/pola-rs/polars/pull/5761. It is also included in the #Python #Polars 0.15.3 release: https://github.com/pola-rs/polars/releases/tag/py-0.15.3

Publications

• Christina Taylor who leads the Next Generation Communications Platform at Carvana recently published the blog “Open Format in an Omni-cloud World: From EDW targets to Structured Streaming on Delta Lake” which showcases her team’s migration from a data warehouse (Big Query) to Delta Lake using structured streaming.
• Simon Whiteley and I had our monthly Ask Us Anything answering your most pressing Delta Lake questions https://youtu.be/2HxXM150TnA
• Vini Jaiswal, Florian Valeye, and Will Girten did a 2022 retrospective in the last community office hours for the year on Delta Spark #apachespark, Delta Rust #rustlang, and Delta Sharing #deltasharing respectively. https://youtu.be/4OK7jpzj5yM
• At Linux Foundation’s Open Source in Finance Forum, Antoine Amend and Ashley Trainer discuss the FINOS Legend project as part of the session?Modernize Regulatory Reporting: Get Ready for T+1 Settlement where they include a call out for Delta Lake to harmonize with open data standards.

Call out to Delta Sharing

A shout out to Will Girten for this great graphic showcasing all of the integrations with Delta Sharing including Java, Power BI, Terminal, MLflow, node.js, C++, Go, Airflow, Excel, R, and Rust!

• Java connector: https://github.com/databrickslabs/delta-sharing-java-connector
• MLflow: https://github.com/databrickslabs/arcuate
• Node.js: https://github.com/goodwillpunning/nodejs-sharing-client
• Power BI: https://learn.microsoft.com/en-us/power-query/connectors/deltasharing
• GoLang: https://github.com/magpierre/delta-sharing/tree/golangdev/golang/delta_sharing_go
• C++: https://github.com/magpierre/cpp_delta_sharing_client
• Terminal: https://github.com/magpierre/dsmb
• Airflow: https://github.com/apache/airflow/pull/22692
• R: https://github.com/zacdav-db/delta-sharing-r
• Excel: https://www.exponam.com/exponam-launches-delta-sharing-excel-add-in/
• Rust: https://github.com/r3stl355/delta-sharing-rust-client

Connect with us!

Want to learn more about Delta Lake and chat with other users and contributors? Join us at?delta.io,?Slack,?LinkedIn, and?GitHub.