Spotlight on Databricks Model Serving and Lakehouse Monitoring

AI is a whole new world, and there’s a whole new dictionary to go with it. To read my future articles, join my network by clicking 'Follow'.

--------------------------------------------------------------------------------

SO MANY announcements around Databricks new #AI capabilities!?

Why so many announcements? The truth is that it’s not easy for companies to build and manage LLM apps. Databricks has been creating lots of tools, so that our customers are able to build delightful LLMs customized with their own proprietary data.

In my last blog, I summarized three announcements from the week of Dec 4:?

• Suite of RAG tools (original blog)
• Public Preview of Vector Search?(original blog)
• Public Preview of Feature & Function Serving (original blog)

Today’s blog will summarize key points from the following two announcements. (Yes more announcements.) We dive deep into more RAG tools.

• Databricks Model Serving. This section covers Custom Models that you create yourself, Foundation Model APIs (with flexible pricing), and External Models (formerly called AI Gateway).
• Lakehouse Monitoring. A simple dashboard that tells you everything you need to know. Comes with pre-configured profiles like Snapshot or Time Series.

Databricks Model Serving

Imagine you have a universal remote control. It can operate all kinds of electronic devices in your house, from your TV to your sound system. Databricks has updated its Model Serving tool. And it now acts like a universal remote. It's a single remote that lets you easily access different GenAI models from different sources.?

You can access any Foundation Model. This includes:

• Custom Models
• Databricks-managed models
• Third-party Foundation models

It does not matter where they are hosted. It also does not matter whether they are provided by Databricks or other companies. So, it’s like using a single remote for devices from different brands.

Custom Models

You can create your own original AI model. And you can use your own proprietary data set to tailor it to your specific needs. Then, when it’s ready, the Databricks Model Serving can deploy it and manage it.

(Let’s review from my Chapter 3 blog, what is model serving again? After training a chatbot (or LLM), model serving is how you make the results of the model available to others.)

Databricks offers tools and an environment for your custom AI model. You can train and fine-tune models with your private data. This improves the accuracy and relevance of your model. Databricks uses techniques such as RAG (Retrieval Augmented Generation) and PEFT (Parameter-Efficient Fine-tuning). And it’s all integrated with Databricks Vector Search. These tools make it easier to use and scale up as needed.

Foundation Model APIs

Foundation Model APIs allows you to access popular open source LLMs.

API is the key word. These Foundation Model APIs are connectors provided by Databricks. They allow you to easily access and work with large, pre-trained AI models. You can incorporate these models into your applications or projects without the need for complex setup or infrastructure management.

Think of an API as like a waiter in a restaurant. You (the developer) are sitting at a table, and you want to order food (data or information). The waiter (the API) takes your order (your request) and goes to the kitchen (the Foundation Model) to get it. Then, the waiter brings you your food (the data or information) on a plate.

The restaurant in this case is Databricks. The food served is information from a Databricks managed model like MPT-7B or MPT-30B .

In addition, the Foundation Model API’s can access other open source models, like Llama2. Check out Databricks Marketplace for the latest list of models.

Flexible Pricing

The best part? You only pay for what you eat! If you only order a small bite (a few words), you pay a little. If you order a big feast (a lot of text), you pay more. This makes it flexible and affordable for everyone.

Zero hefty upfront cost. Foundation Model APIs can be used on a pay-per-token basis. Alternatively, they can be used on a fixed capacity (provisioned throughput). These options? significantly reduces operational costs.?

External Models (formerly known as AI Gateway)

External Models allow you to access proprietary 3rd party Foundation Models. It works by adding an “endpoint” for models hosted outside of Databricks.?

Imagine you have a fancy coffee machine in your kitchen (Databricks). You can make all sorts of coffee drinks with it, like lattes, cappuccinos, and espressos.

But sometimes, you want a special drink, like a matcha latte or a Vietnamese iced coffee. Different tools and ingredients are needed for these drinks. Your coffee machine cannot make them by itself.

This is where endpoints come in! An endpoint is like a special adapter for your coffee machine. It lets you connect to other machines or services (external models) that can make those special drinks.

So, you can plug in an endpoint for a matcha latte maker. And then use your coffee machine to make a delicious matcha latte! You can have multiple endpoints for different drinks. This gives you access to more options and variety.

And the best part? You can control everything from your coffee machine (Databricks). You can choose which endpoint to use, adjust the settings, and even monitor the progress of your drink.

With External Models, you can add endpoints to powerful proprietary AI models. These include Azure OpenAI, Anthropic Claude, AWS Bedrock Models, and AI21 Labs task specific models.?

Other things you can do with Databricks Model Serving:

Query Models via a Unified Interface

This allows you to compare different models from a single interface. It does not matter that they are from different sources. Then, you can switch models easily. And you can combine different models like Legos to create even more amazing AI models.?

Govern and Monitor All Models

This is like your control center. A centralized UI simplifies security. You can manage permissions, track usage, and monitor quality for all models. This includes external models.

You decide who can use which tools. You can set usage limits. And you can monitor quality.??

Lakehouse Monitoring

Lakehouse Monitoring, the second announcement this week, is like having a security camera system for your data.?

Security cameras in a house allow you to see if everything is safe. Lakehouse Monitoring lets you check on your data , features, and ML models.?

It helps you make sure that everything is up-to-date and working right. This is helpful for people who handle large amounts of data and large models. You can identify and resolve issues quickly.

And Lakehouse Monitoring is integrated with Unity Catalog. So, you can monitor quality together with governance. This gives you? deep insight into the performance of your data and AI assets.

Last but not least, Lakehouse Monitoring is Serverless. This means that it takes care of all the boring IT stuff. No need to set-up servers and install applications. So you can focus on what's important: making the most of your data and AI.

How it works

It’s a dashboard! Ta-da!!! How simple is that?

• One-Click. It’s a one-click dashboard that is super user friendly. It gives users an immediate and complete view of the data quality in their systems. They can see various metrics and visualizations that represent the quality and health of their data.
• Automatic Computation of Metrics. The system automatically calculates different statistics about your data. This includes basic statistical measures. It also includes distributional metrics and model performance metrics, like accuracy for ML models.
• Custom Metrics. If you have specific things you want to check in your data, you can set up your own measurements. This lets you focus on what's important for your work.
• Storage in Delta Tables. All these measurements are saved in Delta tables. This makes it easy to look back at your data over time and do more detailed analysis if you need to. Users can conduct deeper analyses beyond what is available in the standard dashboard. They can explore trends, identify anomalies, or correlate different data points.

Configuration Options

You can choose from several monitoring profiles. These are blueprints for setting up how you want to track specific aspects of your data and AI models. Think of them as pre-defined configurations or recipes for monitoring. They save you time and effort.

Pick any table inside Unity Catalog. Then start monitoring snapshots, time series, or ML model quality over time.

1. Snapshot Profile. This option is for monitoring the entire data set over time. It's useful if you want to compare current data with previous versions or a baseline. The system will calculate metrics across all data in the table every time it refreshes. This is particularly cool for tracking long-term trends. And it is helpful for understanding how your data evolves over time.
2. Time Series Profile. This profile helps you compare data over different time periods. It includes timestamps like event dates and times. You can compare data hourly, daily, or weekly. It's especially useful for understanding patterns and trends over time. If you enable Change Data Feed, you get the benefit of incremental updates each time the monitor refreshes. This is efficient and timely.
3. Inference Log Profile. This is tailored for monitoring machine learning models. It helps you track model performance over time and see how inputs and predictions change. If your table includes inputs and outputs from an ML model, this profile will be very beneficial. Additionally, you can include metadata. For example, ground truth labels for drift analysis. And demographic information for fairness and bias metrics.

You have the choice to determine the frequency of the monitoring service. This can be daily, hourly, or another frequency. This flexibility ensures that the monitoring aligns with your specific data needs. It also aligns with your specific business processes.

Regular check-ups

Scan your data and AI regularly. Then create reports that show how healthy they are over time. Think of it like taking your temperature or checking your pulse.

These reports come in two flavors:

• Profile Metrics (kinda like a Report Card). This shows basic stats like how many blank spaces or zeros are in your data. Or, how accurate is your model?. It's a snapshot of their health.
• Drift Metrics (kinda like a comparison test). This sees how your data has changed over time.

Set up alerts

These are like little alarm bells that ring when something goes wrong.?

• For data tables: They'll yell if there are too many empty spaces or zeros.
• For models: They'll warn you if they're not working well anymore.

With alerts, you can catch bad data before it ruins your results. You can fix the problems early.

And you can make your models better. You can retrain them if they're not doing their job right.

Monitoring your LLMs

Lakehouse Monitoring offers a full solution to ensure the quality of your RAG apps. It automatically checks the outputs of your RAG for potential issues like:

• Toxic content. Identifies harmful or offensive language in your model's responses.
• Safety concerns. Detects responses that might be inaccurate, misleading, or biased.
• Data errors. Flags issues related to outdated or incorrect information used by the model.
• Model behavior. Helps diagnose unexpected or inconsistent behavior in your model's performance.

Catch issues early before they impact your users or reputation. Lakehouse Monitoring simplifies the process of maintaining your RAG applications. It ensures high quality and reliability. This should give you peace of mind and confidence in their outputs.

About the author: Maria Pere-Perez

This is my last blog this year. Happy holidays, y'all! See you next year!!!

The opinions expressed in this article are my own. This includes the use of analogies, humor and occasional swear words. I currently work as the Director of ISV Technology Partnerships at Databricks. However, this newsletter is my own. Databricks did not ask me to write this. And they do not edit any of my personal work. My role at Databricks is to manage partnerships with AI companies, such as Dataiku, Pinecone, LangChain, LlamaIndex, Posit, MathWorks, Plotly, etc... In this job, I'm exposed to a lot of new words and concepts. I started writing down new words in my diary. And then I thought I’d share it with people. Click "Subscribe" at the top of this blog to learn new words with me each week.

You can see my past blogs here.