The Databricks Drop - 2023-05-29

Here are some of the interesting resources/articles that have come from Databricks in the last couple of weeks

???ML & AI

• New Solution Accelerator!?Enhancing Product Search with LLMs?This example utilises?Wayfair's annotation dataset (WANDS)?to fine-tune a?SentenceTransformer('all-MiniLM-L12-v2')?model, generate and store embeddings in a vector store (Chroma), register the model and embeddings in MLflow registry, and finally deploy the model for serving (accompanying blog post)
• New Solution Accelerator!?Build your own Chatbot: walks through indexing documents, generating embeddings (using OpenAI embeddings), persisting embeddings in a vector store (FAISS), creating a Q&A flow (using Langchain), persisting the model in MLflow registry, and serving the model for your applications (Accompanying blog post)
• Databricks????Hugging Face: The new?from_spark?function in Hugging Face reduces data processing time by over 40%. It combines Spark's efficiency with Hugging Face's pipeline integration, aiding in AI model optimisation
• Actioning Customer Reviews at Scale with Databricks SQL AI Functions: bring meaning to unstructured data using the simplicity of SQL and GPT-3.5

???Retail

• Simplify entity resolution with Databricks Automated Record Connector (ARC)?ARC abstracts away the complexity of utilising UK Ministry of Justice's?Splink library?for entity resolution. It determines the optimal set of blocking rules, comparisons, and deterministic rules (GitHub repo)

???Training & Education

• Enroll in our New Expert-Led Large Language Models (LLMs) Courses on edX?starting Jun 8! Master Large Language Models with expert guidance, hands-on learning, and insights from industry pioneers. Explore cutting-edge techniques like prompt engineering, embeddings, vector databases, and model tuning. Learn from luminaries like?Stanford Professor & Databricks Co-Founder Matei Zaharia?and the creators of Dolly. Access free course materials to audit and elevate your LLM expertise. Consistent with our goal of keeping things open, course materials are?free?for anyone to audit.?Enroll today

User Experience

• Test out the new navigation UI?and give us your feedback. The unified navigation bar has been a great time-saver, allowing me to switch between screens in fewer clicks

????Developer Experience

• 30+ reusable Terraform modules to provision your Databricks Lakehouse platform: the newly released?Terraform Registry modules for Databricks?provides 30+ reusable Terraform modules and examples to provision your Databricks Lakehouse platform. I've found this useful for exploring best practices and speeding up Terraform development
• New debugging features for Databricks Notebooks with Variable Explorer: a welcome productivity boost for debugging! Variable Explorer allows you to inspect the value of any variable or dataframe as you step through your code. Combine this with?pdb?to set breakpoints and investigate interactively with UI support in the Databricks notebook.
• ???Databricks Connect v2 Quickstart

???Data Engineering

• How Databricks improved query performance by up to 2.2x by automatically optimizing file sizes: Unity Catalog managed tables takes care of optimising file sizes for you, so you don't have to worry about it. Instead you can sit back and enjoy the performance boost!
• Processing data simultaneously from multiple streaming platforms using Delta Live Tables: walkthrough of simultaneously ingesting and transforming streams across Azure Event Hubs, Kafka, and Kinesis
• Track health and fitness goals with Apple Healthkit and Databricks: great walkthrough of building out health insights using a metadata-driven approach with Delta Live Tables for ETL (GitHub repo)

???Data Analysis / SQL

• Using Databricks SQL in VSCode
• Actioning Customer Reviews at Scale with Databricks SQL AI Functions: bring meaning to unstructured data using the simplicity of SQL and GPT-3.5
• Predictive I/O for Reads?is now GA! It's an ML feature, to enhance the speed and efficiency of point lookups. By intelligently predicting required data and eliminating the need for costly indexes or optimization services, it significantly improves query performance. Real-world applications have demonstrated up to 35x faster response times compared to other cloud data warehouses, all while reducing complexity and maintenance overhead
• Introduction to Databricks SQL: a good walkthrough by one of our partners on the capabilities of Databricks SQL

???Performance Optimisation

• Understanding Caching in Databricks SQL: UI, Result, and Disk Caches?
• Predictive I/O for Optimised Reads?is now generally available for SQL Pro and Serverless. Provides the benefits of indexes and optimization services, but without the complexity and cost of maintaining them.
• Latency goes subsecond in Apache Spark Structured Streaming: progress has been made on?Project Lightspeed, and the introduction of async offset management shows a?3x improvement?in stream latency

???Security

• Welcome Okera: Adopting an AI-centric approach to governance: Databricks will integrate Okera's capabilities into Unity Catalog, including AI-powered discovery and handling of sensitive (e.g. PII) data and new isolation technology for workloads.
• Cluster Policy Primer: Cluster policies allow you to enforce security and cost controls on new clusters. This post walks you through what cluster policies are and how to design them for your Workspaces
• The Hitchhiker's Guide to data privilege model and access control in Unity Catalog

???Databricks Marketplace

• Public Preview of Databricks Marketplace: an open marketplace for all your data, analytics, and AI needs. Unlike traditional data marketplaces, providers can also share notebooks, models, and dashboards. Access both free (e.g. Carto's spatial features for Australia) and paid data. Check it out by clicking on the Marketplace icon at the bottom of your left-hand navigation menu

As always, please let me know if you'd like to find out more about any of the announcements or use cases above ????