The Databricks Drop - 2023-09-07

?? Events, Training amp; Education

• ?? Data+AI World Tour ANZ!: I want to extend an invite to our in-person Data+AI World Tour events happening in Sydney and Brisbane!We'll have talks from practitioners at Endeavour, Atlassian, NAB, SportsBet, Red Cross, Seven West Media, Tabcorp, and more! Besides learning about what's new in the world of Databricks, this is a fantastic opportunity to meet and mingle with peers in the industry. Please sign up at the below links, and I'm looking forward to seeing you there!
• Sydney Tue 19 & Wed 20 Sep: https://vnjv.co/daiwt23syd (Day 1 is in-person training and certification) // Brisbane Tue 14 Nov: (link coming soon)
• Data + AI Summit Session videos & slides are now available on-demand!
• Introducing "Ask Databricks": Your Direct Line to Our Product Experts: a new live weekly series for Databricks Users and Data Practitioners co-hosted with our partners Advancing Analytics . Although the timing isn't great for those of us in Australian timezones, you can catch replays on Databricks' and Advancing Analytics' YouTube channels. ??? Schedule

???? Best Practices

• NEW ? Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads: a new guide walking you through all the possible ways to tune your workloads for ?? performance and ?? cost
• NEW ? Demo Hub: access ready-to-run Notebooks that walk you through common worfklows and use cases on Databricks
• NEW ? What's coming page: provides an overview of what's coming in near future releases, helping you plan ahead of time

?? ML & AI

• NEW ? Demo: Deploy Your LLM Chatbot With Retrieval Augmented Generation (RAG) and llama2-70B (MosaicML inferences): this end-to-end demo showcases building a RAG-based chatbot, and showcases Databricks Vector Search, MLflow AI gateway, and MosaicML inference (access the ?? notebooks directly here). While some of these capabilities are in preview, it gives you a taste of what's coming!
• NEW ? Solution Accelerator: Automated Analysis of Product Reviews Using Large Language Models (LLMs)(?? Notebooks)
• NEW ? GitHub repo with detailed Notebooks on how to work with OSS LLMs (e.g. Llama-2-*, mpt-*) using MLflow in Databricks
• Managing Complex Propensity Scoring Scenarios with Databricks(Notebooks| GitHub): a new Solution Accelerator to manage regular feature updates (utilising Feature Store) and periodic model re-training
• Announcing the MLflow AI Gateway: an API gateway to govern enterprise use of Saas (e.g. OpenAI, Bard, Anthropic) and OSS LLMs. With MLflow AI Gateway you can:Manage API keys in a central locationManage costs by enforcing quota and rate controlsImprove developer productivity by exposing a standard API interface across all model providers
• The Simplification of AI Data: a summary of the all the latest with Feature Store and integration with Unity Catalog

?? Retail

• NEW Solution Accelerator: Automated Analysis of Product Reviews Using Large Language Models (LLMs)(?? Notebooks)
• Go from Months to Hours with Databricks Marketplace for Retailers

??? Developer Experience

• Introducing Lakehouse Apps: build secure native data and AI applications without compromise. Are you building applications on top of your Lakehouse data? Lakehouse Apps allows you to build and host those applications directly on your Lakehouse, preventing the need to move data around and worrying about inconsistent governance models.

?? Data Engineering

• Never Miss a Beat: Announcing New Monitoring and Alerting capabilities in Databricks Workflows: Duration warnings alert you when jobs take longer than usual to execute. This allows you to identify and rectify issues sooner rather than later! Fine-grained notifications allow you to control the types of alerts (e.g. failure, duration warning) each recipient receives
• Delta UniForm: a universal format for lakehouse interoperability: Delta UniForm allows for your Delta tables to be recognised by Iceberg and Hudi readers without the need to convert your tables. You're able to maintain a single copy of your data, continue benefiting from Delta Lake's performance, and achieve greater interoperability with the rest of your data and AI stack
• Modular Orchestration with Databricks Workflows: modular jobs now allow you to define Workflows with jobs-within-jobs

?? Security

• How PII scanning is done at Seek using Presidio and Unity Catalog (a great solution put together by Ajmal Aziz !)
• Identifying and Tagging PII data with Unity Catalog

?? Customer Stories

• Australian Red Cross Lifeblood wins 2023 Data for Good Award: Red Cross are able to attract more donations and strengthen community bonds through initiatives that span granular forecasting, real-time wait-time predictions, customer segmentation, and marketing attribution
• How Akamai Leverages Databricks Unity Catalog For Distributed Data Governance (accompanying Summit presentation and slides): with 50 exabytes of data accessed across 80 workspaces, Unity Catalog brought many efficiencies to the team:Eliminated the need for duplicating mounts for shared datasets across workspacesImplemented fine-grained access controls at the row and column levelsEliminated the pains of syncing user management and access control across multiple workspacesSingle pane of glass for data observability
• Accelerating Innovation at JetBlue ?? Using Databricks: (?? video) thanks to Databricks JetBlue has benefited with: Rapid prototyping, iteration, and launching of data pipelines, jobs, and ML models; Elevated customer experience; Continuous improvement of overall value; Lowered TCO
• FactSet's Lakehouse adoption resulted in faster processing times and improved team productivity while reducing costs by 83%

As always, please let me know if you'd like to find out more about any of the announcements or use cases above ????